Release Management
Space shortcuts
Space Tools

Versions Compared

Old Version 124

changes.mady.by.user Reeta Metta

Saved on

compared with

New Version Current

changes.mady.by.user Jeffrey Klann

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Info Box
titleImportant links
  • User registration tool: 6.5a i2b2 Webclient User Registration
    • SAML

    Downloads:

    Documentation:


    Warning
    titleIssue Loading CRC Stored Procedures on MSSQL

    Note that we have discovered an issue loading the CRC Stored Procedures on SQL Server and have found a workaround. Please be sure to read the updated content of Step 3.4.4 Create Crcdata Stored Procedures before performing this step.

    1.7.13 Release Notes

    i2b2 1.7.13 offers support for SAML federated login, enhanced security due to improvements found via an internal Veracode scan, a client-based user registration tool, support for Synthea synthetic data for testing, and a variety of other bugfixes and performance improvements. i2b2 1.7.13 has been tested with SHRINE 3.2.1.  

    UI Button
    colorred
    sizesmall
    iconstar
    titleSHRINE Approved
    urlhttps://open.med.harvard.edu/wiki/display/SHRINE/SHRINE-i2b2+Compatibility+Matrix

    Table of Contents

    Highlight of Features

    ...

    Description
    SAML Authentication

    Image RemovedImage Added

    User Account Registration Tool

    local registrationImage Modified

    ACT Ontology v4

    Image Modified

    Improved patient counting scripts
    ("totalnum")

    Image RemovedImage Added

    Synthea SyntheticMass dataset in i2b2 format

    Image RemovedImage Added

    Simplified database upgrade method

    Image Modified

    log4J upgrade (to address security concerns)

    Apache Log4j Logo.pngImage Modified

    Code changes to address security vulnerabilities

    Image Modified

    Bugfixes

    Image Modified

    Community-Contributed Features

    Contribution

    Contributor


    SAML Authentication

    Kevin Bui (lead developer)
    Michelle Morris
    University of Pittsburgh

    Justin Prosser (security expert)
    University of Washington

    Mike Mendis
    Jeff Klann
    Reeta Metta
    Mass General Brigham

    		i2b2 now includes support for SAML-based enterprise authentication via an institutional Identity Provider. See more information below.


    Ability to specify user parameter defaults

    Michael Horvath
    Wake Forest University

    This change is meant to allowing allow user params to take precedence over hive params. Currently, it's the other way around.

    Particularly, if you have the situation where you have a large number of users who use an authentication method other than the default basic, but your service account (AGG_SERVICE_ACCOUNT) is using basic then you need to specify a user param for each of your users.

    With this change, you can set default authentication params in pm_hive_params for all users, and then set your service account specifically to be authentication_method = 'BASIC'. May seem like a small change, but it would be very welcome not to need to assign params for each new user at our institution.
    In the process of making this change, I re-organized basic authentication into its own package and removed some code duplication. 

    LDAP UPN Support

    Michael Horvath
    Wake Forest University

    Active Directory enables other methods of binding which are more flexible besides just using the distinguished name. https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/6a5891b8-928e-4b75-a4a5-0e3b77eaca52. This change is to enable binding the the User Principle Name form, which is very convenient when the distinguished names for users is not easily available (OU by department, etc.).

    API to get all children of an ontology node

    Kevin Bui
    Michelle Morris
    University of Pittsburgh

    The metadata GetChildren API call, which returns information on the children of an ontology node, can now be configured to return multiple levels of children (e.g., children, children's children, etc.). This is done by specifying the numLevel parameters. 

    By default, the function assumes numLevel = 1 and will return the direct descendants of the concept, which is one level of children.  When the numLevel = -1 the function will return ALL descendants of the concept, otherwise the function will return up to and including the number of levels specified by numLevel (eg. numlevel=2 returns two levels of descendants, numLevel=4 returns four levels of descendants).

    Totalnum Counter Performance ImprovementsDarren Henderson
    University of Kentucky    		Performance enhancements on SQL Server totalnum counting to not : stop unnecessarily recompute recomputing temp tables.
    ACT v4 Postgres bugfixes

    Ambreen Zaver
    University of Colorado

    		Bugfixes in time interval calculation (for age and age-at-visit) in ACT v4 ontology for Postgres.

    ...

    There is a new user registration tool that can be enabled in the webclient. It allows users to request an i2b2 account that can then be activated by an administrator. It supports manual entry of user information through the form shown below, or automatic population of user information through SAML.

    Documentation on this new feature is here: 6.5a i2b2 Webclient User Registration

    local registrationImage Modified

    SAML Authentication

    Image Removed

    i2b2 now includes support for SAML-based enterprise authentication via an institutional Identity Provider.

    Detailed setup instructions are in Chapter 8 of the Installation Guide.

    Image Added

    Improved Totalnum Scripts

    Totalnum Scripts (patient counting scripts) have been updated to improve the counter's performance on both many multiple ontology tables and very large(>1.5 million) ontologies  such as ACT medications. Debug messages have also been added for troubleshooting and profiling purposes. Support for multiple fact tables has been added and bugfixes have been made.

    Totalnum Scripts Setup

    1. If upgrading, create the totalnum and totalnum_report tables. In Release_1-7/Upgrade/Metadata, run the ant upgrade script.
      ant -f data_build.xml upgrade_tables_release_1-7-12a
    2. In  In the Release_1-7/NewInstall/Metadata/ run the ant script to create the stored procedures. 
      ant -f data_build.xml create_metadata_procedures_release_1-7 
    3. Set privileges: If using multiple schemas, the stored procedure should be run from the metadata schema. Make sure the stored procedure can read the tables in the crcdata schema (observation_fact, visit_dimension, patient_dimension) and can both read an update ontology tables in the metadata schema (including table_access). 
    4. If using multiple fact tables, the recommended approach is to create a fact table view as the union of all your fact tables. (This is essentially going back to a single fact table, but it is only used for totalnum counting. This is needed to correctly count patients that mention multiple fact tables within a hierarchy.)
          e.g., 
       create view observation_fact_view as
       select * from CONDITION_VIEW 
       union all
       select * from drug_view
      
If running the counting script in SQL Server, add the wildcard flag, to ignore multifact references in the ontology:
      e.g. exec RunTotalnum 'observation_fact_view','dbo','@','Y'
      This is automatically accounted for in the other database platforms.
Note this approach does not work if you have conflicting concept_cds across fact tables.
    5. Run the stored procedures on your database. This can be done in two ways:
      • Run the ant command to execute the data_build.xml file with below specified target 
        POSTGRESQL : ant -f data_build.xml db_metadata_run_total_count_postgresql
        ORACLE : ant -f data_build.xml db_metadata_run_total_count_oracle
        SQL SERVER : ant -f data_build.xml db_metadata_run_total_count_sqlserver   
      • Execute the RunTotalNum  stored procedure manually against your database
      in
      • from a
      sql
      • SQL Client. This can take several hours
      .  Example is
      • for large databases or large ontologies.  Examples are below.
       

    Running the totalnum stored procedure directly


    Oracle:     

    ...

    begin
        RUNTOTALNUM('observation_fact','i2b2demodata');
    end; 

    ...

    You can optionally include a table

    ...

    name if you only want to count one ontology table (this IS case sensitive):
    begin
      runtotalnum('observation_fact','i2b2demodata','I2B2');
    end;

    Note: If you get the error as: ERROR at line 1: ORA-01031: insufficient privilege, then run the command:
            grant create table to (DB USER)  

    SQL server: 

    ...

    exec RunTotalnum 'observation_fact','dbo','@'
    Parameters are: 1) the observation table name (for multi-fact-table setups), 2) the

    ...

    schema name, 3) a single table name to run on a single ontology table or '@' to run on all, and 4) and a wildcard flag that will ignore multifact references in the ontology if 'Y'
    PostgreSQL: 

    ...

     

    ...

    ...

    select RUNTOTALNUM('observation_fact','public')
    Replace 'public' by the schema name for the fact table
    If using a schema other than public for metadata, you might need to run "set search_path to 'i2b2metadata','public' " first as

    ...

    well

    When finished, verify it is complete by checking that c_totalnum columns in your ontology tables contain numbers (not nulls). These total counts will be visible in the ontology browser in the web client.

    ...

    Parent folders will get counts (of all patients with facts in the leaves) except for ontology folders derived from visit_dimension or patient_dimension. These cannot be rolled up because of the way these terms are defined in the ontology. They will have no count at all (not a zero).

    Info

    i2b2 users must have the DATA_AGG user permission to view the counts through the web client.

    Additional New Stored Procedures

    Age In Years Updater

    When the CRC data is installed via ant, a new SQL script updates the age_in_years_num in the patient dimension based on the birth datedates of the sample patients. As a reminder, this load process can be triggered with ant -f data_build.xml db_demodata_load_data in the CRC directory of NewInstall.

    Concept Dimension Updater

    Insert_Concept_FROMTableAccess  is designed to populate concept_dimenison table using Tablethe ontologies listed in table_access table records.
    The stored procedure loops through the table_access  and  and inserts values from corresponding each metadata table (specified in the c_table_name metadatatable which have column), when
    c_tablename column value as dimtablename is set to 'concept_dimension'
    Example usage: exec Insert_Concept_FROMTableAccess

    I2b2-Synthea data Load

    Synthetic patient data generated by Synthea can be loaded into i2b2. The Synthea SyntheticMass sample files have been converted to i2b2-ACT format, and scripts to load Synthea data from scratch are available here: https://github.com/i2b2/i2b2-synthea

    Synthea Load Process

    ...

    1. Set up an i2b2 project with the ACT ontology.
    2. Either download the SyntheticMass 63k sample in i2b2 format from https://github.com/i2b2/i2b2-synthea/blob/main/syntheamass_63K_sample.zip, or follow the instructions below to load any Synthea dataset from scratch. This information can also be found on the Synthea-i2b2 Community Project page.

    Loading Synthea data from scratch
    1. Download SyntheticMass Data, Version 2 (24 May, 2017) 
      • All data sets (1k, COVID 10k, COVID 100k) have been verified to work EXCEPT the 100k patients in the large SyntheticMass Version 2 download. 
      • The 100k patients in the large SyntheticMass Version 2 download needs an extra step to delete invalid records before import. In this case, download synthea_cleanup.pl to your disk, and then run "synthea_cleanup <directory-for-synthea-csv-files>" The fixed csv files will be in <directory-for-synthea-csv-files>/fixcsv.2.
    2. Set up an i2b2 an i2b2 project with the ACT ontology.
    3. Download the scripts from https://github.com/i2b2/i2b2-synthea)
    4. Run create_synthea_table_<your dbServertype>.sql in your project to create the Synthea tables.
    5. Import the Synthea data you downloaded in step one into the Synthea tables in your project.
    6. Load the i2b2-to-SNOMED table in this repository into your project. https://www.nlm.nih.gov/healthit/snomedct/us_edition.html
    7. Click on the "Download SNOMED-CT to ICD-10-CM Mapping Resources" link to download. (You will need a UMLS account.)
    8. Unzip the file
    9. Import the TSV file into a table called SNOMED_to_ICD10 in your database.
    10. In Postgres and Oracle, follow the additional instructions in the comments at the top of synthea_to_i2b2_<your dbServerType>.sql to clean up the date formatting.
    11. Run synthea_to_i2b2_<your dbServertype>.sql to convert synthea Synthea data into i2b2 tables (this will truncate your existing fact and dimension tables!)
    12. Replace references to i2b2metadata.dbo in the script. Use the database and schema where your ACT ontology tables are.

    ACT Version-4 Ontology data load

    Metadata scripts are now available to load the latest ACT Version-4  Ontology 4 ontology into your i2b2 db schema.

    Note

    The CPT4 ontology table is not included with i2b2 due to AMA restrictions on redistribution of CPT code information. Contact the ACT team to get a copy if your institution is an AMA member.

    ACT4 data load process

    ...

    1. Download and extract the newinstall  newinstall zip package from "Download Binary Distribution" in the top section of https://www.i2b2.org/software/download.html?d=452
    2. Extract the  metadata\act folder from the downloaded zip folder
    3. Replace
    4. Edit the  edu.harvard.i2b2.data\Release_1-7\NewInstall\Metadata\act  folder with extracted new act folderEdit the  db.properties file in your metadata folder  file  to update the project properties to 'ACT' ; db.project=ACT
    5. From the edu.harvard.i2b2.data\Release_1-7\NewInstall\Metadata folder, run the ant command: ant -f data_build.xml db_metadata_load_data
      1. This will execute the  SQL scripts from the edu.harvard.i2b2.data\Release_1-7\NewInstall\Metadata\act\scripts\<db type> folder and create and load ACT4 Ontology metadata
      tables  
      1. tables 
    6. You can now verify the new Ontology  by logging into the webclient.

    ...

    1. i2b2 has been made more secure by addressing parameterization and other potential vulnerabilities found in an internal a Veracode scan.
    2. Log4J has been upgraded to the latest version. Following The following jars are updated in lib/axis2.war/WEB-INF/lib the folder:
      • log4j-api-2.17.1.jar
      • log4j-core-2.17.1.jar
      • log4j-jcl-2.17.1.jar

    Improved db Upgrade Process

    Currently Previously, i2b2 db upgrade is was a multi-step process of running upgrade SQL scripts and stored procedures individually on the db instance. This release simplifies the process of running the  the table upgrade SQL scripts and stored procedures from a the  data_build.xml .

    Steps to Perform db upgrade:

    1. Backup your existing data folder
    2. Copy all the folders from the extracted latest version download data folder   into your existing  data Upgrade folder. This will replace existing Crcdata, Hivedata, Metadata, PMdata folders. Example:  Downloads\i2b2core-upgrade-1713\i2b2\data to C:\opt\edu.harvard.i2b2.data\Release_1-7\Upgrade\.
    3. Copy the db.properties files from your back up into the respective locations (namely Crcdata, Hivedata, Metadata, PMdata )
    4. Open the command prompt and navigate  to each cell folder and run the following upgrade ant commands on your i2b2 database instance, where {db} can  be Oracle, sqlserver or postgresql.
      Alternative to above Step, you can run individual SQL scripts on your db instance in place of  ant commands.

    ...

    In  data folder\Release_1-7\Upgrade\  run the ant commands under each individual cell folder as below.

    ...

    1.7.09c

    ...

    In the Crcdata folder run the following ant command: ant -f data_build.xml upgrade_tables_release_1-7-09c upgrade_tables_release_1-7-10 upgrade_tables_release_1-7-11 upgrade_tables_release_1-7-12a

    In the Hivedata folder run the following ant command: ant -f data_build.xml upgrade_tables_release_1-7-11 upgrade_tables_release_1-7-12a

    In the Metadata folder run the following ant command: ant -f data_build.xml upgrade_tables_release_1-7-10  upgrade_tables_release_1-7-12a

    ...

    1.7.10

    ...

    In the Crcdata folder run the following ant command: ant -f data_build.xml upgrade_table_release_1-7-10 upgrade_tables_release_1-7-11 upgrade_tables_release_1-7-12a

    In the Hivedata folder run the following ant command: ant -f data_build.xml upgrade_tables_release_1-7-11 upgrade_tables_release_1-7-12a

    In the Metadata folder run the following ant command: ant -f data_build.xml upgrade_tables_release_1-7-10 upgrade_tables_release_1-7-12a

    1.7.11

    ...

    In the Crcdata folder run the following ant command: ant -f data_build.xml upgrade_tables_release_1-7-11  upgrade_table_release_1-7-12a

    In the Hivedata folder run the following ant command: ant -f data_build.xml upgrade_tables_release_1-7-11 upgrade_tables_release_1-7-12a

    In the Metadata folder run the following ant command: ant -f data_build.xml upgrade_tables_release_1-7-12a

    ...

    In the Crcdata folder run the following ant command: ant -f data_build.xml upgrade_tables_release_1-7-12a

    In the Hivedata folder run the following ant command: ant -f data_build.xml upgrade_tables_release_1-7-12a

    In the Metadata folder run the following ant command: ant -f data_build.xml upgrade_tables_release_1-7-12a

    Finally,  upgrade your db instance( if Postgres) with stored procedures. Details are here

    files. Details are on the i2b2 Upgrade Page here.

    Changelog

    Database Drivers

    The JDBC drivers were updated to the following versions.

    Driver

    ojdbc8.jar

    postgresql-42.2.

    5

    14.jar

    mssql-jdbc-

    7

    9.

    0

    2.0.jre8.jar

    New Version

    Oracle 21.5

    PostgreSQL 42.3.2

    MS Sql Server 9.2

    Supported Db Server versions

    ...

    Supported software versions

    Application Type

    Java

    Wildfly

    Apache HTD

    Apache Ant

    Apache Axis2

    PHP

    Supported Version/s

    8 or 11

    17.0.

    0

    1Final

    2.

    4.17 or higher

    0 (RHEL 6) and 2.2 ( RHEL 7)

    		1.9.61.7.17.2.27 or higher

    Supported Operating Systems

    ...

    CentOS versions 6 (deprecated) or 7 (highly recommended) 

    Windows 7-2019

    Unofficially, MacOS and other flavors of Linux are likely to work.

    i2b2 Server and Client Changes

    New Features and Improvements

    Core-serverwebclient 

    CORE-399 Oracle index hints must use the table alias 

    CORE-382 Username / password errors should not specify which had the problem

    CORE-402 Fix Veracode identified Security flaws in i2b2 Server-Side Code

    CORE-404 Adding support for JDK 11. Now including the gensrc due to jaxb has been removed. 

    CORE-413 FetchAllChildren 

    CORE-414 SAML (server side)

    CORE-415 Log4j upgraded to v2

    CORE-416 User parameter precedence change (contributed by Michael Horvath)

    CORE-417 LDAP UPN support (contributed by Michael Horvath)

    CORE-405 Upgrade JDBC Drivers 

    CORE-412 Disable login to agg service account

    WEBCLIENT-334 Provide tabs for major plugins and temporal query 

    WEBCLIENT-344 Cleanup Analysis Tools list of Plugins to only Supported Items 

    WEBCLIENT-325 Wayne's improvements to hierarchical find-by-name

    WEBCLIENT-353 SAML and user registration tool (client side) 

    i2b2 Database Changes

    New Features and Improvements

    DATA-7 QT_PATIENT_SET_ENC_COLLECTION should be a bigint

    DATA-14 Synthea i2b2 data 

    DATA-6 improve i2b2 db upgrade process

    DATA-12  Stored procedure to update concept dimension

    DATA-11 Age in years updater during demodata install

    DATA-9 ACT v4 ontology

    DATA-13 Postgres time interval corrections in ACT v4 demographics ontology

    CORE-389 Totalnum performance improvements 

    CORE-394 Obfuscated totalnum reporting table

    CORE-398 Multifact support for totalnums  

    CORE-400 Show totalnums in top level folders 

    Bug Fixes

    Webclient Core-server

    WEBCLIENT-351 Obfuscated User Not Showing Graph

    WEBCLIENT-342 switch response status check from "OK" to 200 to handle lab value pop up in http/2 protocols

    WEBCLIENT-335 temporal query in webclient with no anchoring events not running

    WEBCLIENT-350  Unable to drag items in workplace 

    WEBCLIENT-294 Webclient Reports "QUERY CANCELLED" While Query Is Still Running 

    WEBCLIENT-354 Removed broken context menu in Find Previous Queries

    CORE-418 Local timestamps in sessions on Oracle - appserver /db server in diff time zones 

    CORE-282 Error returned when obfuscated user is locked out 

    CORE-281 Query continues to run after user receives lockout message 

    CORE-399 use alias for index hint  

    Notes for Developers

    For Java 11 install, if you change the xsd (REST API message definitions), then you will need to regenerate gensrc via JAXB in Java 8. In the i2b2-core cell directory for which you're regenerating the XSD-Java, run the ant target target "jaxb_gen" on Java 8 and then build as usual using Java 11.

    ...