Transformers have been extraordinarily successful in modelling language and a range of other phenomena. When trained on sufficiently large data and using self-supervised learning, these transformers are known as foundation models.
We want to industrialize the development of foundation models across domains. sistemalabs is the foundation model factory.
We publish sequifier, the library for the development of foundation models. It implements the standard causal, autoregressive transformer architecture, with configurable architecture specification, input and target variables, loss functions, and more. It is designed to scale across GPUs and soon across nodes.
Our focus is on training state/action foundation models, that model sequences of states and actions. We interpret 'action' liberally, to include people (clicks, purchases, transactions), animals (behaviour, communication), cells, neural activity, markets and robotics.
Many interesting data sets of this type are proprietary. This is why we are happy to partner with companies to jointly develop foundation models in their domain, for internal use or public release.
sequifier
Sequifier implements a unified workflow for training single- or multivariate causal decoder-only transformer models, and preprocessing the data into the appropriate format. The idea is: implement it once, use it everywhere, make it configurable to cover as many scenarios as possible.
applications
-
Stripe: A transaction foundation model for fraud detection
Stripe trains models on transaction sequences to learn transaction history embeddings. These embeddings enable the detection of various kinds of fraud at significantly higher rates than previous machine learning systems.
Read article → -
Siemens: A timeseries foundation model for predictive maintenance
Siemens uses a transformer model to learn the relationships of sensor values over time. The model can then be used for forecasting and for retrieval of 'qualitatively' similar time series that occurred in the past.
Read transcript → -
UC Berkeley/Agility Robotics: Humanoid locomotion as next token prediction
Researchers apply generative modeling to robotics by training transformers to predict the next sensorimotor token. This allows a humanoid to navigate real-world cities zero-shot, effectively by treating physical movement like language.
Read paper →
library
We provide a curated library of scientific publications that are relevant to the sistemalabs mission. It's not meant to be exhaustive, it's just papers we like <3
offerings
- enterprise partnership
Support for your organization in developing and deploying a sequifier foundation model.
- academic support
Scientific applications of sequifier get support at preferential rates.
- phd students
PhD students (and earlier) that want to use sequifier for a research project can apply for a fellowship.
- contract foundation model development
Give us the data and requirements, and we train a foundation model end-to-end. You own the weights.