MIP, my proposal for a high-performance analysis pipeline for whole exome sequencing

Hi all,

It has been a while since my last post, but the reason is worth the long time absent. Since January I am co-leading the bioinformatics and I.T department of one of Genomika Diagnósticos. 

Genomika is one of most advanced clinical genetics laboratory in Brazil. Located in Recife, Pernambuco, in the Northeast of Brazil, it provides cutting edge molecular testing in cancer samples to better define treatment options and prognosis, making personalized cancer management a reality. It also has a vast menu of tests to evaluate inherited diseases, including cancer susceptibility syndromes and  rare disorders. Equipped with state-of-the-art next-generation sequencing instruments and a world-class team of specialists in the field of genetic testing, Genomika focus on test methods that improve patient care and have immediate impact on management.  There is available a pitch video about our lab and one of our exams (unfortunately the spoken language is portuguese).

Our video about sequencing exams spoken in portuguese

My daily work is to provide tools; infra-structure and systems to support our clients and teams in the lab. One of major teams is the molecular biology sector. It is responsible for the DNA sequencing exams, which includes targeted-panels, specific genes or exons or whole exome.  Each one of those genetic tests, before delivered to the patient and the doctor,  goes under several data pre-processing and analysis steps organised in a ordered set of sequential steps, which we call a pipeline.

There is a customised pipeline for clinical sequencing; where we bioinformaticians and specialists study the genetic basis of human phenotypes. In our lab pipeline we are interested on selecting and capturing the protein-coding portion of the genome (we call the exome).  This region responsible for only 3% of our human DNA can be used to elucidate the genetic causes of many human diseases, starting from single gene disorders and moving on more complex genetic disorders, including complex traits and cancer.

Clinical Sequencing Pipeline overview

For this task,  we use several tools that must handle with large volumes of data, specially because of the new next-generation DNA sequencing machines (yeah we have one at our lab from Illumina). Those machines are capable of producing in shorter times and lower costs  large amount of NGS data. 

Taking those challenges into account, we perform our sequencing, alignment, detection and data analysis of human samples in order to seek variants.  This study we call variant analysis. Variant analysis looks for variant information, that is, possible mutations that may be associated to genetic diseases . Let's consider as examples of mutation or variant as follows: a change of nucleotide (A for T) (single nucleotide variant or SNV) or even a small insertion or deletion (INDEL's) that can impact the functional activity of the protein.  Looking after variants and even further seek and identify those related to diseases or genetic disorders is a big challenge in terms of technology, tools and interpretation.

The reference in the genome at bottom; the variants above.
In this example there's a possible exchange of A to G (SNV) in a specific position of the genome.

In our lab we are developing a streamlined, highly automated pipeline for exome and targeted panel regions data analysis. In our pipeline we handle multiple datasets and state of the art tools that are integrated in a custom pipeline for generating, annotating and analyzing sequence variants.

We named our internal pipeline tool as MIP (Mutation Identification pipeline). Some minimal requirements we stablished for MIP in order to use it with maximum performance and productivity. 

1. It must be automatic;  with a limited team like ours (2 or 3 bioinformaticians) we need a efficient service that is capable to execute the complete analysis without typing commands at terminals calling software or converting files among several data formats.

MIP pipeline overview for clinical sequencing. All those steps requires tools and files in a specific format.
Our engine   would be capable of manage and execute all or some of those steps with
specific parameters defined by the specialist .

2. It must be user-oriented; it mean that MIP platform must provide an easy-to-use interface, that any researcher of lab could use the system and start out-of-box their sequencing analysis.  For biologists and geneticists it would allow them to focus their work on what matters: the downstream experiments.

3. Scalable-out architecture;  More and more hight throughput sequencing data is pulled out from NGS instruments, so MIP must be designed to be a building block for a scalable genomics infrastructure. It means that we must work with distributed and parallel approaches and the best practices from high-performance computing and big data to efficient take advantage of all resources available at our infra-structure while thinking on continuous optimization in order to minimize the network and shared disk I/O footprint.

My draft proposal to our exome sequencing pipeline

4. Rich-detailed reports and smart software and dataset updates;  In order to maintain our execution engine working healthy, it requires that our software stack always being updated. Since our engine is written on top of numerous open-source biological and big data packages, we need a self-contained management system that could not only check for any new versions but also with a few clicks start any update and perform a post-check for any possible corruptions at the pipeline.  In addition to the third-party genomics software used on MIP, we are also developing our tool for variant annotation. So it stands for an engine that could query and analyze several genomic dataset, generate real-time interactive reports where the researchers could filter out variants based on specific criteria and output in formats of QC reports, target and sequencing depth information, descriptions of the annotations and variants hyperlinked to public datasets in order to get further details about a variation.

Example of web interface where a researcher could select any single or combination of annotations to display. Links to the original datasources are readily available (Figure from WEP annotation system)

5. Finally, we think the most important requirement nowadays to MIP is the integration with our current LMS (Laboratory management system), in order to put the filtered variants  as input to our existing laboratory report analysis and publishing workflow. It means more productivity and automation with our existing infrastructure.

MIP could be also be acessible via RESful API, where the runs output

 would be interchanged with our external LMS solution.

As you may see, there's a huge effort on coding, design and infrastructure to meet those requirements. But we are thrilled to make this happen!  One of our current works in this project is the genv tool. Genv is what we call our genomika environment builder. The basic idea behind it is a tool written in python and fabric package, that provides instant access to biological software, programming libraries and data. The expected result  is a fully automated infrastructure that installs all software and data required to start MIP pipeline.  We are thinking of also arranging pre-built images with Docker.  Of course I will need a whole post to explain more about it!

To sum up,  I hope I could summarise one of the projects I've been working this first semester. At Genomika Diagnósticos we are facing big scientific challenges and the best part is that those tools are helping our lab to provide a next level of health information to the patients,  all from our DNA!

If you are interested on working with us, keep checking our github homepage with any open positions at our bioinformatics team.

Until next time!

Deepjewel - Social Media powering Recommendations

Hi all,

Happy new year!  My first post this year will be about an idea that I had with my friend Ricardo Caspirro about the next generation of social recommenders in commerce applications and retail stores. What excites me is that this idea came from a conversation that we had in 2009, and since that year we discussed more about what it would be the "Deepjewel".

Deepjewel is a giant knowledge base that encapsulates interesting entities and relationships of the social world in the web.  The social world in this context means all the millions and billions of tweets, Facebook messages, profiles, relationships, blog posting, YouTube videos, and more - a living organism itself, constantly evolving. 

The Deepjewel

But what motivated us to create the Deepjewel?  One of main problems that we face nowadays is the discovery of content and items of our interest.  Many times, for instance, to find a book or a movie that we like, it is required to search at several websites and social networks through the web.  There isn't a tool that allow this connection between items of many domains in a organized and structured way, even for easy access. Those objects are spread over the web, and the recommendations are placed in social networks by comments, results of machine learning techniques or by queries at several web pages or search engines.  The problem becomes worst when we don't know anything about the existence of a certain item, which it could result at never finding out that possible item that would be of our interest. 

The social media is huge and we need tools that performs a deep analysis of all this data, filter out items of my interest, specially from the historical data  (with our permission, of course)  from our presence in the web and bring items and products relevant to us without loosing the discovery process associated with the serendipity.  One of the solutions is a powerful recommender engine fed by this Social Genome.

Hybrid Social Context-Aware Recommender RecDay

This hybrid social context-aware recommender (which we call recday) is a engine composed by several modular components, which we employ a broad range of semantic analysis techniques, including information extraction and integration, natural language processing and machine learning. The main task of this recommender is to analyze information about his posts, bios and relationships/lists collected from the social genome and summarize it (all this data would represent the interests of the user) in profiles, which we could call DNA. Those profiles built by the recday would infer the possible interests of the users and would serve as basis for personalized recommendations of products and services from the retail stores/e-commerce applications.

A perfect example for this proccess, which we call the translation, is when you mentioned several times about Apple products (such as macbook, ipod, iphone, etc) at your tweets. Even you never used the word "Apple", we can use the Social Genome to detect the products and infer that you are interested in Apple products. The following figure illustrate certain kinds of entities and relationships collected in the Social Genome:

The relationships extracted from the social web data

The second step of this engine is to build the user profile. Different from another approaches which it only uses the content or the historical data from reviews or ratings from the user, the Recday would go further and would analyze the temporal context included in the interests of the user. Several reports on consumer behavior show that the user desires are influented by external factors and even the humour or feelings of the person at the certain moment. It is required to collect in a stealth way (with the user permission of course) his social data and build his personality defined by several dimensions. Those dimensions represent the current state of the user which may define what kind of suggestions he would like to receive at that particular moment. For example, if you are happy today because you got a new job and posted at your Facebook about that event or even updated your profile about this new position, it would be a valuable information for your DNA profile in order to recommend products and services to celebrate this occasion (You are happy and excited, don't you think ?).

Another important component in this proccess is the product side. We need extract more information from their products portfolio. Items must be juiced in order to get all its meta-data available. Imagine the movie Batman where we have details about the year, genre, cast, production, direction, sinoypsis, etc. All this data can be used to build the DNA Item and be expressed by a collection of dimensions that represent the item profile. With those profiles (DNA User and DNA Item) we compute the similarity between user and items in order to produce a list of top recommendations and related explanations.

The Social Architecure of the Recommender

The final result can be shown in several mediums: mobile apps, widgets, web, API's, pluggins, etc. It is important the usability and how you will present all this information for a particular user. That's why the user interface must be simple and easy to navigate and have mechanisms to collect the user's feedback for the suggestions given by the engine. This proccess is cyclic, so when you give a feedback (a like or dislike or a comment about the suggestion), this piece of information is handled and leveraged to power your DNA profile.

A particular medium of the recommendations: Ipad Demo

In order to build all these interesting technical challenges, we needed to start developing our in-house solution called Crab, which proccesses all this data and employ several analysis and filtering techniques  to deal with the percularities of this heterogeneous sources of data. The first start is the launch of the Deepjewel Labs. Deepjewel is a principle that we can mine the wealth in data, by identifying interesting entities and relationships and converting them into valuable information as input in the recommendation proccess of items and services. 

In summary, all those human and computation techniques can be used to perform a deep semantic analysis of web and social data, where the result for a commerce application or retail store is the ability to offer what the users want before they know that they really want in a personalized way. The RecDay  would be able to daily offer relevant product and services to their customers without they even know it exists. This is a new way to shop in which you don't have to go find products, service and information; the machine will help them find their way to you.

To know more about our Deepjewel labs, visit the website (currently in portuguese):  

http://www.deepjewel.com

I hope you enjoyed,

Marcel Caraciolo

Introducing the Architecture REST - Creating APIs - Part 02

In this post, we continue talking about the REST architecture and our progress to create our first API.  This is the second part of the series, if you want to see the Introduction, please check this link. 

The SOA Principles

The REST is also considered a Service-Oriented Architecture (SOA) , so it the SOA principles must be considered. The REST architecture follows the interoperability by providing support for several types of response. It is a good practice to make your resource accept HTML, JSON or XML in the same method, turning it more portable.  Depending of the target of your application, the number of clients of your services can be bigger in this way. However, providing all those types of response consumes time, and you have to analyze if you are available for working on this interoperability.

It is also important that your RESTful services don't take overloaded operations. This includes heavy processing tasks such as data transformations (XML to JSON, XML to Text/Plain, etc.) as also the validation of the data before each transformation.  This results in the other principle of SOA the weak coupling, that is, make your system less dependent of other modules in order to reduce the modification effects and failure tailoring.  Generally the simple RESTful services systems are divided in the following packages:

• resources -  The resources of your system which implement the HTTP methods POST, GET, PUT, DELETE. For each request received, it uses the modules of the utils package and the communication provided with dao, if necessary. The resources also have the task of interpret the result , possible failures and response to the client.
• utils - The utility classes as also related to data transformation.
• dao -  The classes with the pattern DAO (Data Access Object), responsible for the database transactions

It is also common to see the developers abuse of the methods POST and GET.  Using REST, the developer must know well the four main methods: POST, GET, PUT and DELETE.  We will explain in the next paragraphs about those and comment a little about the HEAD and OPTIONS. For illustrate the explanation, we will use a resource user implemented at http://www.example.com/resources/user/.

POST

Submit the data to be processed at a target resource, which can result in the creation of a new or an update of the existing resources. The data are included in the body of the request. For instance, making a request POST to http://www.example.com/resources/user/  and including the body represented by the JSON in the Figure 01, we will requesting to the server to add this new user. If the user was created successfully, the response will be with a status code 201, pointing that our resource was created.

Figure 1 - JSON as body

GET

Used to request a new representation of the resource specified. In practice, making a request GET in http://www.example.com/resources/user/10 will return as response the user whose the id is 10.  In the other hand if our request was to the url http://www.example.com/resources/user/ , we would have as response the list of all users.

PUT

]It updates the representation of the resource. The data must be sent in the body of the request. Besides, if the URI of the request doesn't not point out to a existing resource, it is allowed to create a new resource with this URI. In our example, if we wanted to update the password of the user with the login "marcelcaraciolo", we just need to make a PUT request to http://www.example.com/resources/user/10 putting the JSON body represented in the next figure, containing the refreshed data.

Figure 2 - JSON as body

DELETE

It deletes a specified resource. It doesn't have the body. If we request the DELETE method to URI http://www.example.com/resources/user/10 it would mean to the serve that we want it to delete the user with the id is equal to 10.

HEAD

Similar to the method GET, but without the body of the response. This method can be used to obtain the metadata of a entity target of the request, without transfer all the data (the body itself of the entity) to the client.

OPTIONS

Return the HTTP methods that the server supports for a specified URI. It can be used to check the features available of a web service. Making a request OPTIONS to http://www.example.com/resources/user/ , we would receive the attribute 'Allow' in the headers with the fields OPTIONS and POST. However making the request OPTIONS to the URI http://www.example.com/resources/user/* ,  we would receive the response OPTIONS, GET, PUT, DELETE and HEAD.  But wait, you may be asking yourself , where is the POST ? Is it missing ?  When you put the wildcat  (*) it is expected some response, but our method POST, using the good practices, is not mapped to accept requests with the URI finishing in 'user/*'.  In other words,  it doesn't make sense to request a POST to 'user/10' , since the id of the resource must be created by the serve. Besides that, in a OPTIONS request, in the body it will come the WADL file, which we will discuss later.

It is important to API developers to know that the client of the service represents the user. So, it is a convention to not use the methods GET and HEAD to do actions,  except the action of information recovering, therefore they are considered safe methods.

Although, knowing that the methods GET and HEAD are secure, the user are conscious about the fact that an action possibly insecure will be requested if it uses the other HTTP methods. To sum up, don't use GET and HEAD to make requests that generate collateral effects. So use the GET method to modify an entity is an anti-pattern and not a good practice in developing REST APIs.

Another important property is the idempotence. PUT, DELETE and OPTIONS are idempotent methods, that is, multiple operations must always return the same result and have the same effect in the application as  one. For instance, making several GET requests to the same URI, it will always return the same response, in case of the requested data didn't change in the interval between the requests.  Finally, the safe methods are indempotent, since it doest not present collateral effects.

HTTP Status Codes

The HTTP Status Codes were created to allow the developers to describe precisely for the clients what happened at the server or even have control of the services. For that, the more specified the response, better. 

The Status Codes has those meanings:

• 1.xx -  Information
• 2.xx - Success
• 3.xx -  Redirect
• 4.xx -  Client Error
• 5.xx - Server Error

It is important to developers to know how to response properly with your services. Typically, the responses used by the REST services are 200, 404 or 500, but it is important to understand better the responses of other RESTful services that your application are consuming.  We will see some statuses  codes as follows.

200 - OK

The status code 200 represents that the request was successful. The response returned depends on the method used in the request. If the GET method is used, it will returned the entity corresponding to the requested resource.  In case of POST method, the headers will correspond to the requested resources. Finally if the method was HEAD, it will be returned the headers fields that correspond to the requested resources either.

201 - Created

The request was accomplished and the location of the resource created is returned by the field 'Location' in the response headers. The server must create the resource before return the status code 201. If the resource can't be created immediately, the server must response with 202 (Approved) instead.

202 - Accepted

The request was approved, but it doesn't mean that was finalized. Your purpose is to allow the server to accept asynchronous requests.  Depending on the scenario, it is interesting to include to the body of the response the current status of the request and a path to a state monitor or a time estimative to the request be accomplished.

204 - No Content

The server accomplished the request  successfully but it does not need to response any entity in the body.  Optionally, it may include new or updated meta-information about the the entities in the headers. The response 204 must not have the body. Generally it is the status code response to a DELETE request.

304 - No Modified

If the client has requested with a conditional GET method, but  the documents or the requested data weren't modified, the server must response with this status code. The response 304 must not have a body.

400 -  Invalid Request

Invalid Syntax in the request.

401 -  Not Authorized

The request requires authentication or the user has been refused by the credentials provided.

404 -  Not Found

The server didn't find the resources that correspond to the URI of the request. Generally this response comes as result of a GET request.

409 -  Conflict

There was a conflict in the request with the current state of the resource. This code is only allowed in situations where it expects that the user is able to solve the the conflict and re-send again the request.  For that, the body of the response must include an error message. Generally this scenario occurs in responses to PUT requests.

500 - Internal Server Error

The server came into a unexpected condition that stopped it of finishing the request. For instance, if a problem occurred with the database connection, the response must have the status code 500.

The table below provided gives you a resume of the most used http statuses codes when you are developing your RESTful services.  Please read carefully and know how to use it correctly in order to help the developers that will build applications that will consume your services and know how to handle properly the responses of your API.

HTTP protocol version 1.1 Server Response Codes

In the next post about the REST Services I will present a new service that I am developing using the concepts explained in this series of posts.  It will be related to Location + Question and Answers + Mobile + Python with real code of course! 

I hope you enjoyed,

Marcel Caraciolo

References

[1] SOA Principles

[2] http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

[3] HTTP Protocols