0 | Green Deal Data Observatory

dataset: Create Interoperable FAIR Datasets

Thu, 11 Aug 2022 00:00:00 +0000

Interoperable, FAIR datasets

The primary aim of dataset is create well-referenced, well-described, interoperable datasets from data.frames, tibbles or data.tables that translate well into the W3C DataSet definition within the Data Cube Vocabulary in a reproducible manner. The data cube model in itself is is originated in the Statistical Data and Metadata eXchange, and it is almost fully harmonzied with the Resource Description Framework (RDF), the standard model for data interchange on the web[^1].

A mapping of R objects into these models has numerous advantages:

Makes data importing easier and less error-prone;
Leaves plenty of room for documentation automation, resulting in far better reusability and reproducability;
The publication of results from R following the FAIR principles is far easier, making the work of the R user more findable, more accessible, more interoperable and more reusable by other users;
Makes the placement into relational databases, semantic web applications, archives, repositories possible without time-consuming and costly data wrangling (See From dataset To RDF).

Our package functions work with any structured R objects (data.fame, data.table, tibble, or well-structured lists like json), however, the best functionality is achieved by the (See The dataset S3 Class), which is inherited from data.frame().

Contact

For contact information, contributors, see the package homepage.

Code of Conduct

Please note that the dataset project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

statcodelists: Use Standardized Statistical Codelists

Tue, 28 Jun 2022 00:00:00 +0000

Retrospective data harmonization

The aim of retroharmonize is to provide tools for reproducible retrospective (ex-post) harmonization of datasets that contain variables measuring the same concepts but coded in different ways. Ex-post data harmonization enables better use of existing data and creates new research opportunities. For example, harmonizing data from different countries enables cross-national comparisons, while merging data from different time points makes it possible to track changes over time.

Retrospective data harmonization is associated with challenges including conceptual issues with establishing equivalence and comparability, practical complications of having to standardize the naming and coding of variables, technical difficulties with merging data stored in different formats, and the need to document a large number of data transformations. The retroharmonize package assists with the latter three components, freeing up the capacity of researchers to focus on the first.

Specifically, the retroharmonize package proposes a reproducible workflow, including a new class for storing data together with the harmonized and original metadata, as well as functions for importing data from different formats, harmonizing data and metadata, documenting the harmonization process, and converting between data types. See here for an overview of the functionalities.

The new labelled_spss_survey() class is an extension of haven’s labelled_spss class. It not only preserves variable and value labels and the user-defined missing range, but also gives an identifier, for example, the filename or the wave number, to the vector. Additionally, it enables the preservation – as metadata attributes – of the original variable names, labels, and value codes and labels, from the source data, in addition to the harmonized variable names, labels, and value codes and labels. This way, the harmonized data also contain the pre-harmonization record. The stored original metadata can be used for validation and documentation purposes.

The vignette Working With The labelled_spss_survey Class provides more information about the labelled_spss_survey() class.

In Harmonize Value Labels we discuss the characteristics of the labelled_spss_survey() class and demonstrates the problems that using this class solves.

We also provide three extensive case studies illustrating how the retroharmonize package can be used for ex-post harmonization of data from cross-national surveys:

The creators of retroharmonize are not affiliated with either Afrobarometer, Arab Barometer, Eurobarometer, or the organizations that designs, produces or archives their surveys.

We started building an experimental APIs data is running retroharmonize regularly and improving known statistical data sources. See: Digital Music Observatory, Green Deal Data Observatory, Economy Data Observatory.

Citing the data sources

Our package has been tested on three harmonized survey’s microdata. Because retroharmonize is not affiliated with any of these data sources, to replicate our tutorials or work with the data, you have download the data files from these sources, and you have to cite those sources in your work.

Afrobarometer data: Cite Afrobarometer Arab Barometer data: cite Arab Barometer. Eurobarometer data: The Eurobarometer data Eurobarometer raw data and related documentation (questionnaires, codebooks, etc.) are made available by GESIS, ICPSR and through the Social Science Data Archive networks. You should cite your source, in our examples, we rely on the GESIS data files.

Citing the retroharmonize R package

For main developer and contributors, see the package homepage.

This work can be freely used, modified and distributed under the GPL-3 license:

citation("retroharmonize")
#> 
#> To cite package 'retroharmonize' in publications use:
#> 
#>   Daniel Antal (2021). retroharmonize: Ex Post Survey Data
#>   Harmonization. R package version 0.1.17.
#>   https://retroharmonize.dataobservatory.eu/
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title = {retroharmonize: Ex Post Survey Data Harmonization},
#>     author = {Daniel Antal},
#>     year = {2021},
#>     doi = {10.5281/zenodo.5006056},
#>     note = {R package version 0.1.17},
#>     url = {https://retroharmonize.dataobservatory.eu/},
#>   }

Contact

For contact information, contributors, see the package homepage.

Code of Conduct

Please note that the retroharmonize project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

Identifying Roadblocks to Net Zero Legislation

Tue, 16 Mar 2021 00:00:00 +0000

In our use case we are merging data about Europe’s coal regions, harmonized surveys about the acceptance of climate policies, and socio-economic data. While the work starts out from existing European research, our retroharmonize survey harmonization solution, our regions sub-national boundary harmonization solution and iotables allows us to connect open data and open knowledge from other coal regions of the world, for example, from the Appalachian economy.

Policy Context

The Just Transition Platform aims to assist EU countries and regions to unlock the support available through the Just Transition Mechanism. It builds on and expands the work of the existing Initiative for Coal Regions in Transition, which already supports fossil fuel producing regions across the EU in achieving a just transition through tailored, needs-oriented assistance and capacity-building.

The Initiative has a secretariat that is co-run by Ecorys, Climate Strategies, ICLEI Europe, and the Wuppertal Institute for Climate. While the initiative is an EU project, it cooperates with other similar initiatives, for example, with the Coalfield Development social enterprise in the Appalachian economy.

Data Sources

Coal regions: Our starting point is the EU coal regions: opportunities and challenges ahead publication Joint Research Centre (JRC), the European Commission’s science and knowledge service. This publication maps Europe’s coal dependent energy and transport infrastructure, and regions that depend on coal-related jobs.
Harmonized Survey Data: The dataset of the Eurobarometer 91.3 (April 2019) harmonized survey. Our transition policy variable is the four-level agreement with the statement More public financial support should be given to the transition to clean energies even if it means subsidies to fossil fuels should be reduced (EN) and Davantage de soutien financier public devrait être donné à la transition vers les énergies propres même si cela signifie que les subventions aux énergies fossiles devraient être réduites (FR) which is then translated to the language use of all participating country.
Environmental Variables: We used data on pm and SO2 polution measured by participating stations in the European Environmental Agency’s monitoring program. The station locations were mapped by Milos to the NUTS sub-national regions.

Exploratory Data Analysis

Our coal-dependency dummy variable is base on the policy document Coal regions in transition.

readRDS(file.path("data", "coal_regions.rds"))

## # A tibble: 253 x 5
##    country_code_is~ region_nuts_nam~ region_nuts_cod~ coal_region is_coal_region
##    <chr>            <fct>            <chr>            <chr>                <dbl>
##  1 BE               Brussels hoofds~ BE10             <NA>                     0
##  2 BE               Liege            BE33             <NA>                     0
##  3 BE               Brabant Wallon   BE31             <NA>                     0
##  4 BE               Antwerpen        BE21             <NA>                     0
##  5 BE               Limburg [BE]     BE22             <NA>                     0
##  6 BE               Oost-Vlaanderen  BE23             <NA>                     0
##  7 BE               Vlaams Brabant   BE24             <NA>                     0
##  8 BE               West-Vlaanderen  BE25             <NA>                     0
##  9 BE               Hainaut          BE32             <NA>                     0
## 10 BE               Namur            BE35             <NA>                     0
## # ... with 243 more rows

Our exploratory data analysis shows that respondent in 2019, agreement with the policy measure significantly differed among EU member states and regions.

transition_policy <- eb19_raw %>%
  rowid_to_column() %>%
  mutate ( transition_policy = normalize_text(transition_policy)) %>%
  fastDummies::dummy_cols(select_columns = 'transition_policy') %>%
  mutate ( transition_policy_agree = case_when(
    transition_policy_totally_agree + transition_policy_tend_to_agree > 0 ~ 1, 
    TRUE ~ 0
  )) %>%
  mutate ( transition_policy_disagree = case_when(
    transition_policy_totally_disagree + transition_policy_tend_to_disagree > 0 ~ 1, 
    TRUE ~ 0
  )) 

eb19_df  <- transition_policy %>% 
  left_join ( air_pollutants, by = 'region_nuts_codes' ) %>%
  mutate ( is_poland = ifelse ( country_code == "PL", 1, 0))

Preliminary Results

Significantly more people agree where

there are more polutants
who are younger
where people are more educated

Significantly less people agree

in rural areas
where more people are older
where more people are less educated
in less polluted areas
in coal regions

A simple model run:

c("transition_policy_totally_agree" , "pm10", "so2", "age_exact", "is_highly_educated" , "is_rural")

## [1] "transition_policy_totally_agree" "pm10"                           
## [3] "so2"                             "age_exact"                      
## [5] "is_highly_educated"              "is_rural"

summary( glm ( transition_policy_totally_agree ~ pm10 + so2 + 
                 age_exact +
                 is_highly_educated + is_rural + is_coal_region +
                 country_code, 
               data = eb19_df, 
               family = binomial ))

## 
## Call:
## glm(formula = transition_policy_totally_agree ~ pm10 + so2 + 
##     age_exact + is_highly_educated + is_rural + is_coal_region + 
##     country_code, family = binomial, data = eb19_df)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -1.7690  -1.0253  -0.8165   1.2264   1.9085  
## 
## Coefficients:
##                      Estimate Std. Error z value Pr(>|z|)    
## (Intercept)        -0.1975096  0.0921551  -2.143 0.032095 *  
## pm10                0.0068505  0.0017445   3.927 8.60e-05 ***
## so2                 0.1381994  0.0405867   3.405 0.000662 ***
## age_exact          -0.0075018  0.0007873  -9.529  < 2e-16 ***
## is_highly_educated  0.2953905  0.0311127   9.494  < 2e-16 ***
## is_rural           -0.1277983  0.0313321  -4.079 4.53e-05 ***
## is_coal_region     -0.2624005  0.0640233  -4.099 4.16e-05 ***
## country_codeBE     -0.3290891  0.0916117  -3.592 0.000328 ***
## country_codeBG     -0.6470116  0.1125114  -5.751 8.89e-09 ***
## country_codeCY      0.8471483  0.1273306   6.653 2.87e-11 ***
## country_codeCZ     -0.5754008  0.0965974  -5.957 2.57e-09 ***
## country_codeDE      0.0106430  0.0856322   0.124 0.901088    
## country_codeDK      0.0577724  0.0925391   0.624 0.532429    
## country_codeEE     -0.8041188  0.0989047  -8.130 4.28e-16 ***
## country_codeES      1.1266903  0.0941495  11.967  < 2e-16 ***
## country_codeFI     -0.2617501  0.0946837  -2.764 0.005702 ** 
## country_codeFR      0.0130239  0.1639339   0.079 0.936678    
## country_codeGB      0.2454631  0.0891845   2.752 0.005918 ** 
## country_codeGR      0.2169278  0.1209199   1.794 0.072816 .  
## country_codeHR     -0.1632727  0.1001563  -1.630 0.103064    
## country_codeHU      0.5779928  0.1020987   5.661 1.50e-08 ***
## country_codeIT     -0.1427249  0.0940144  -1.518 0.128985    
## country_codeLU     -0.3111627  0.1140426  -2.728 0.006363 ** 
## country_codeLV     -0.6246590  0.0963526  -6.483 8.99e-11 ***
## country_codeMT      0.3303363  0.1228611   2.689 0.007173 ** 
## country_codeNL      0.1707080  0.0902189   1.892 0.058470 .  
## country_codePL     -0.2843198  0.1228657  -2.314 0.020664 *  
## country_codePT      0.1447295  0.0899079   1.610 0.107452    
## country_codeRO     -0.0479674  0.0930433  -0.516 0.606177    
## country_codeSE      0.4865939  0.0922486   5.275 1.33e-07 ***
## country_codeSK     -0.2427307  0.0964652  -2.516 0.011861 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 30568  on 22401  degrees of freedom
## Residual deviance: 29313  on 22371  degrees of freedom
##   (5253 observations deleted due to missingness)
## AIC: 29375
## 
## Number of Fisher Scoring iterations: 4

summary( glm ( transition_policy_agree ~ pm10 + so2 + age_exact +
                 is_highly_educated + is_rural, 
               data = eb19_df, 
               family = binomial ))

## 
## Call:
## glm(formula = transition_policy_agree ~ pm10 + so2 + age_exact + 
##     is_highly_educated + is_rural, family = binomial, data = eb19_df)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -2.1970   0.5035   0.5803   0.6495   0.8465  
## 
## Coefficients:
##                     Estimate Std. Error z value Pr(>|z|)    
## (Intercept)         1.807823   0.079297  22.798  < 2e-16 ***
## pm10                0.005092   0.001239   4.108 3.99e-05 ***
## so2                 0.003274   0.051410   0.064  0.94922    
## age_exact          -0.009781   0.000988  -9.900  < 2e-16 ***
## is_highly_educated  0.396743   0.039735   9.985  < 2e-16 ***
## is_rural           -0.107448   0.037953  -2.831  0.00464 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 20488  on 22401  degrees of freedom
## Residual deviance: 20250  on 22396  degrees of freedom
##   (5253 observations deleted due to missingness)
## AIC: 20262
## 
## Number of Fisher Scoring iterations: 4

Next Steps

After careful documentation, we will very soon publish all the processed, clean datasets on the EU Zenodo repository with clear digital object identification and versioning.
We will seek contact with the Secretariat of the Initiative for Coal Regions in Transition to process all the data annexes in the EU coal regions: opportunities and challenges ahead report.
With our volunteers we want to include coal regions from the United States, Latin America, Australia, Africa first – because we have harmonized survey results – and gradually add the rest of the world.
We will ask political scientists and policy researchers to interpret our findings.

retroharmonize R package for survey harmonization

Tue, 25 Aug 2020 00:00:00 +0000

Retrospective data harmonization

The vignette Working With The labelled_spss_survey Class provides more information about the labelled_spss_survey() class.

In Harmonize Value Labels we discuss the characteristics of the labelled_spss_survey() class and demonstrates the problems that using this class solves.

We also provide three extensive case studies illustrating how the retroharmonize package can be used for ex-post harmonization of data from cross-national surveys:

The creators of retroharmonize are not affiliated with either Afrobarometer, Arab Barometer, Eurobarometer, or the organizations that designs, produces or archives their surveys.

Citing the data sources

Citing the retroharmonize R package

For main developer and contributors, see the package homepage.

This work can be freely used, modified and distributed under the GPL-3 license:

citation("retroharmonize")
#> 
#> To cite package 'retroharmonize' in publications use:
#> 
#>   Daniel Antal (2021). retroharmonize: Ex Post Survey Data
#>   Harmonization. R package version 0.1.17.
#>   https://retroharmonize.dataobservatory.eu/
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title = {retroharmonize: Ex Post Survey Data Harmonization},
#>     author = {Daniel Antal},
#>     year = {2021},
#>     doi = {10.5281/zenodo.5006056},
#>     note = {R package version 0.1.17},
#>     url = {https://retroharmonize.dataobservatory.eu/},
#>   }

Contact

For contact information, contributors, see the package homepage.

Code of Conduct

Please note that the retroharmonize project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

iotables R package for working with symmetric input-output tables

Wed, 03 Jun 2020 00:00:00 +0000

iotables processes all the symmetric input-output tables of the EU member states, and calculates direct, indirect and induced effects, multipliers for GVA, employment, taxation. These are important inputs into policy evaluation, business forecasting, or granting/development indicator design. iotables is used by about 800 experts around the world.

Code of Conduct

Please note that the iotables project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

Market size of the re-usable public sector information in Hungary

Tue, 15 Dec 2009 00:00:00 +0000

Original title in Hungarian: A közintézmények újrahasznosítható információinak piaca Magyarországon.

0 | Green Deal Data Observatory

dataset: Create Interoperable FAIR Datasets

Interoperable, FAIR datasets

Contact

Code of Conduct

statcodelists: Use Standardized Statistical Codelists

Retrospective data harmonization

Citations and related work

Citing the data sources

Citing the retroharmonize R package

Contact

Code of Conduct

Identifying Roadblocks to Net Zero Legislation

Policy Context

Data Sources

Exploratory Data Analysis

Preliminary Results

Next Steps

retroharmonize R package for survey harmonization

Retrospective data harmonization

Citations and related work

Citing the data sources

Citing the retroharmonize R package

Contact

Code of Conduct

iotables R package for working with symmetric input-output tables

Code of Conduct

Market size of the re-usable public sector information in Hungary