Release Management
Space shortcuts
Space Tools

Versions Compared

Old Version 137

changes.mady.by.user Reeta Metta

Saved on

compared with

New Version 138

changes.mady.by.user Jeffrey Klann

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

There is a new user registration tool that can be enabled in the webclient. It allows users to request an i2b2 account that can then be activated by an administrator. It supports manual entry of user information through the form shown below, or automatic population of user information through SAML.

Documentation on this new feature is here: 6.5a i2b2 Webclient User Registration

local registration

SAML Authentication

Image Removed

i2b2 now includes support for SAML-based enterprise authentication via an institutional Identity Provider.

Detailed setup instructions are in Chapter 8 of the Installation Guide.

Image Added

Improved Totalnum Scripts

Totalnum Scripts (patient counting scripts) have been updated to improve the counter's performance on both many multiple ontology tables and very large(>1.5 million) ontologies  such as ACT medications. Debug messages have also been added for troubleshooting and profiling purposes. Support for multiple fact tables has been added and bugfixes have been made.

...

  1. If upgrading, create the totalnum and totalnum_report tables. In Release_1-7/Upgrade/Metadata, run the ant upgrade script.
    ant -f data_build.xml upgrade_tables_release_1-7-12a
  2.  In the Release_1-7/NewInstall/Metadata/ run the ant script to create the stored procedures. 
    ant -f data_build.xml create_metadata_procedures_release_1-7 
  3. Set privileges: If using multiple schemas, the stored procedure should be run from the metadata schema. Make sure the stored procedure can read the tables in the crcdata schema (observation_fact, visit_dimension, patient_dimension) and can both read an update ontology tables in the metadata schema (including table_access). 
  4. If using multiple fact tables, the recommended approach is to create a fact table view as the union of all your fact tables. (This is essentially going back to a single fact table, but it is only used for totalnum counting. This is needed to correctly count patients that mention multiple fact tables within a hierarchy.)
        e.g., 
       create view observation_fact_view as
       select * from CONDITION_VIEW 
       union all
       select * from drug_view
    
If running the counting script in SQL Server, add the wildcard flag, to ignore multifact references in the ontology:
      e.g. exec RunTotalnum 'observation_fact_view','dbo','@','Y'
    This is automatically accounted for in the other database platforms.
Note this approach does not work if you have conflicting concept_cds across fact tables.
  5. Run the stored procedures on your database. This can be done in two ways:
    • Run the ant command to execute the data_build.xml file with below specified target 
      POSTGRESQL : ant -f data_build.xml db_metadata_run_total_count_postgresql
      ORACLE : ant -f data_build.xml db_metadata_run_total_count_oracle
      SQL SERVER : ant -f data_build.xml db_metadata_run_total_count_sqlserver   
    • Execute the RunTotalNum  stored procedure manually against your database from a SQL Client. This can take several hours for large databases or large ontologies.  Examples are below.


Oracle:     

begin
    RUNTOTALNUM('observation_fact','i2b2demodata');
end; 

You can optionally include a table name if you only want to count one ontology table (this IS case sensitive):
begin
  runtotalnum('observation_fact','i2b2demodata','I2B2');
end;

Note: If you get the error as: ERROR at line 1: ORA-01031: insufficient privilege, then run the command:
        grant create table to (DB USER)  

SQL server: exec RunTotalnum 'observation_fact','dbo','@'
Parameters are: 1) the observation table name (for multi-fact-table setups), 2) the schema name, 3) a single table name to run on a single ontology table or '@' to run on all, and 4) and a wildcard flag that will ignore multifact references in the ontology if 'Y'
PostgreSQL:  select RUNTOTALNUM('observation_fact','public')
Replace 'public' by the schema name for the fact table
If using a schema other than public for metadata, you might need to run "set search_path to 'i2b2metadata','public' " first as well

When finished, verify it is complete by checking that c_totalnum columns in your ontology tables contain numbers (not nulls). These total counts will be visible in the ontology browser in the web client. Note that parent

Parent folders will get counts (of all patients with facts in the leaves) except for ontology folders derived from visit_dimension or patient_dimension. These cannot be rolled up because of the way these terms are defined in the ontology. They will have no count at all (not a zero).

...

Insert_Concept_FROMTableAccess  is designed to populate concept_dimenison table using Tablethe ontologies listed in table_access table records.
The stored procedure loops through the table_access  and  and inserts values from each metadata table (specified in the c_table_name column), when
c_dimtablename is set to 'concept_dimension'
Example usage: exec Insert_Concept_FROMTableAccess

...

  1. Download SyntheticMass Data, Version 2 (24 May, 2017) 
    • (All data sets (1k, COVID 10k, COVID 100k) have been verified to work EXCEPT the 100k patients in the large SyntheticMass Version 2 download. 
    • The 100k patients in the large SyntheticMass Version 2 download needs an extra step to delete invalid records before import. In this case, download synthea_cleanup.pl to your disk, and then run "synthea_cleanup <directory-for-synthea-csv-files>" The fixed csv files will be in <directory-for-synthea-csv-files>/fixcsv).
  2. Set up an i2b2 project with the ACT ontology.
  3. Download the scripts from https://github.com/i2b2/i2b2-synthea)
  4. Run create_synthea_table_<your dbServertype>.sql in your project to create the Synthea tables.
  5. Import the Synthea data you downloaded in step one into the Synthea tables in your project.
  6. Load the i2b2-to-SNOMED table in this repository into your project. https://www.nlm.nih.gov/healthit/snomedct/us_edition.html
  7. Click on the "Download SNOMED-CT to ICD-10-CM Mapping Resources" link to download. (You will need a UMLS account.)
  8. Unzip the file
  9. Import the TSV file into a table called SNOMED_to_ICD10 in your database.
  10. In Postgres and Oracle, follow the additional instructions in the comments at the top of synthea_to_i2b2_<your dbServerType>.sql to clean up the date formatting.
  11. Run synthea_to_i2b2_<your dbServertype>.sql to convert synthea Synthea data into i2b2 tables (this will truncate your existing fact and dimension tables!)
  12. Replace references to i2b2metadata.dbo in the script. Use the database and schema where your ACT ontology tables are.

...

Metadata scripts are now available to load the latest ACT Version-4  Ontology 4 ontology into your i2b2 db schema.

Note

The CPT4 ontology table is not included with i2b2 due to AMA restrictions on redistribution of CPT code information. Contact the ACT team to get a copy if your institution is an AMA member.

ACT4 data load process

  1. Download and extract the newinstall  newinstall zip package from "Download Binary Distribution" in the top section of https://www.i2b2.org/software/download.html?d=452
  2. Extract the  metadata\act folder from the downloaded zip folder
  3. Replace
  4. Edit the  edu.harvard.i2b2.data\Release_1-7\NewInstall\Metadata\act  folder with extracted new act folderEdit the  db.properties file in your metadata folder  file  to update the project properties to 'ACT' ; db.project=ACT
  5. From the edu.harvard.i2b2.data\Release_1-7\NewInstall\Metadata folder, run the ant command: ant -f data_build.xml db_metadata_load_data
    1. This will execute the  SQL scripts from the edu.harvard.i2b2.data\Release_1-7\NewInstall\Metadata\act\scripts\<db type> folder and create and load ACT4 Ontology metadata
    tables  
    1. tables 
  6. You can now verify the new Ontology  by logging into the webclient.

...

Improved db Upgrade Process

Currently Previously, i2b2 db upgrade is was a multi-step process of running upgrade SQL scripts and stored procedures individually on the db instance. This release simplifies the process of running the table upgrade SQL scripts  and stored procedures from data_build.xml files. Details are on the i2b2 Upgrade Page here.

Changelog

Database Drivers

...