Skip to content

GitLab

Commit b4e52ea0 authored by Dario Pinto's avatar Dario Pinto
Browse files

Add leo pic, Mohamed in Teams, few blog assets and Format tutorials

parent 96addfa9
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 1636 additions and 0 deletions
scripts/port.sh
View file @ b4e52ea0
......@@ -26,6 +26,9 @@ echo "Create file $name ..."
article='
$0~"<div class=\"entry-links\">"{exit}
p==1 {
line = gensub(/<br>/, "", "g");
line = gensub(/\[code(.*)/, "```", "g");
line = gensub(/\[\/code\]/, "```", "g");
line = gensub(/<p>/,"","g");
line = gensub(/<\/p>/,"\n","g",line);
line = gensub(/<code>([^<]*)<\/code>/,"`\\1`","g",line);
......
src/content/assets/img/photo_leo.png
View file @ b4e52ea0
File suppressed by a .gitattributes entry or the file's encoding is unsupported.
src/content/assets/img/photo_mohamed.jpg 0 → 100644
View file @ b4e52ea0
File suppressed by a .gitattributes entry or the file's encoding is unsupported.
src/content/blog/2013_01_17_beta_release_of_opam.md 0 → 100644
View file @ b4e52ea0
title=Beta Release of OPAM
authors=Louis Gesbert
date=2013-01-17
category=Tooling
tags=beta,release,
OPAM is a source-based package manager for OCaml. It supports
multiple simultaneous compiler installations, flexible package
constraints, and a Git-friendly development workflow. I have recently
announced the beta-release of OPAM on the [caml-list](https://sympa.inria.fr/sympa/arc/caml-list/2013-01/msg00073.html), and this blog post introduces the basics to new OPAM users.
### Why OPAM
We have decided to start writing a brand new package manager for
OCaml in the beginning of 2012, after looking at the state of affairs in
the OCaml community and not being completely satisfied with the
existing solutions, especially regarding the management of dependency
constraints between packages. Existing technologies such as GODI, oasis,
odb and ocamlbrew did contain lots of good ideas that we shamelessly
stole but the final user-experience was not so great — and we disagreed
with some of the architectural choices, so it wasn’t so easy to
contribute to fix the existing flaws. Thus we started to discuss the
specification of a new package manager with folks from [Jane Street](https://www.janestreet.com/) who decided to fund the project and from the [Mancoosi project](https://www.mancoosi.org/)
to integrate state-of-the-art dependency management technologies. We
then hired an engineer to do the initial prototyping work — and this
effort finally gave birth to OPAM!
### Installing OPAM
OPAM packages are already available for homebrew, macports and
arch-linux. Debian and Ubuntu packages should be available quite soon.
In any cases, you can either use a [binary installer](https://ocamlpro.com/pub/opam_installer.sh) or simply install it from [sources](https://github.com/OCamlPro/opam/archive/0.9.1.tar.gz). To learn more about the installation process, read the [installation instructions](https://opam.ocamlpro.com/doc/Quick_Install.html).
### Initializing OPAM
Once you’ve installed OPAM, you have to initialize it. OPAM will store all its state under `~/.opam`,
so if you want to reset your OPAM configuration, simply remove that
directory and restart from scratch. OPAM can either use the compiler
installed on your system or it can also install a fresh version of the
compiler:
```shell-session
$ opam init # Use the system compiler<br>
$ opam init –comp 4.00.1 # Use OCaml 4.00.1<br>
```
OPAM will prompt you to add a shell script fragment to your `.profile`.
It is highly recommended to follow these instructions, as it let OPAM
set-up correctly the environment variables it needs to compile and
configure the packages.
### Getting help
OPAM user manual is integrated:
```shell-session
$ opam –help # Get help on OPAM itself
$ opam init –help # Get help on the init sub-command
```
### Basic commands
Once OPAM is initialized, you can ask it to list the available
packages, get package information and search for a given pattern in
package descriptions:
```shell-session
$ opam list *foo* # list all the package containing ‘foo’ in their name
$ opam info foo # Give more information on the ‘foo’ package
$ opam search foo # search for the string ‘foo’ in all package descriptions
```
Once you’ve found a package you would like to install, just run the usual `install` command.
```shell-session
$ opam install lwt # install lwt and its dependencies
$ opam remove lwt # remove lwt and its dependencies
```
Later on, you can check whether new packages are available and you can upgrade your package installation.
```shell-session
$ opam update # check if new packages are available
$ opam upgrade # upgrade your packages to the latest version
```
Casual users of OCaml won’t need to know more about OPAM. Simply
remind to update and upgrade OPAM regularly to keep your system
up-to-date.
### Use-case 1: Managing Multiple Compilers
A new release of OCaml is available and you want to be able to use it. How to do this in OPAM ? This is as simple as:
```shell-session
$ opam update # pick-up the latest compiler descriptions
$ opam switch 4.00.2 # switch to the new 4.00.2 release
$ opam switch export –switch=system | opam switch import -y
```
The first line will get the latest package and compiler descriptions,
and will tell you if new packages or new compilers are available.
Supposing that 4.00.2 is now available, you can then `switch`
to that version using the second command. The last command imports all
the packages installed by OPAM for the OCaml compiler installed on your
system (if any).
You can also easily use the latest unstable version of OCaml if you want to give it a try:
```shell-session
$ opam switch 4.01.0dev+trunk # install trunk
$ opam switch reinstall 4.01.0dev+trunk # reinstall trunk
```
Reinstalling trunk means getting the latest changesets and recompiling the packages already installed for that compiler switch.
### Use-case 2: Managing Multiple Repositories
Sometimes, you want to let people use a new version of your software
early. Or you are working in a company and expose internal libraries to
your coworkers but you don’t want them to be available to anybody using
OPAM. How can you do that with OPAM? It’s easy! You can set-up your own
repository (see for instance [xen-org](https://github.com/xen-org/opam-repo-dev/)‘s development packages) and add it to your OPAM configuration:
```shell-session
$ opam repository list # list the repositories available in your config
$ opam repository add xen-org git://github.com/xen-org/opam-repo-dev.git
$ opam repository list # new xen-org repository available
```
This will add the repository to your OPAM configuration and it will display the newly available packages. The next time you run `opam update` OPAM will then scan for any change in the remote git repository.
Repositories can either be local (e.g. on your filesystem), remote (available through HTTP) and stored in git or darcs.
### Use-case 3: Using Development Packages
You want to try the latest version of a package which have not yet
been released, or you have a patched version of a package than you want
to try. How could you do it? OPAM has a `pin` sub-command which let you do that easily:
```shell-session
$ opam pin lwt /local/path/
$ opam install lwt # install the version of lwt stored in /local/path
```
You can also use a given branch in a given git repository. For instance, if you want the library `re` to be compiled with the code in the `experimental` branch of its development repository you can do:
```shell-session
$ opam pin re git://github.com/ocaml/ocaml-re.git#experimental
$ opam install re
```
When building the packages, OPAM will use the path set-up with the
pin command instead of using the upstream archives. Also, on the next
update, OPAM will automatically check whether some changes happened and
if the packages needs to be recompiled:
```shell-session
$ opam update lwt # check for changes in /local/path
$ opam update re # check for change in the remote git branch
$ opam upgrade lwt re # upgrade re and lwt if necessary
```
### Conclusion
I’ve briefly explained some of the main features of OPAM. If you want to go further, I would advise to read the [user](https://opam.ocamlpro.com/doc/Advanced_Usage.html) and [packager](https://opam.ocamlpro.com/doc/Packaging.html) tutorials. If you really want to understand the internals of OPAM, you can also read the [developer manual](https://github.com/OCamlPro/opam/blob/master/doc/dev-manual/dev-manual.pdf?raw=true).
src/content/blog/2013_02_18_overview_of_current_activities.md 0 → 100644
View file @ b4e52ea0
title=An Overview of our Current Activities
authors=Çagdas Bozman
date=2013-02-18
category=OCamlPro
tags=OCamlPro,OCaml
From the early days of OCamlPro, people have been curious about our plans; they were asking how we worked at OCamlPro and what we were doing exactly. Now that we have started releasing projects more regularly, these questions come again. They are very reasonable questions, and have resolved to be more public and communicate more regularly. This post covers our activities from the beginning of 2013 and updates are scheduled on a monthly basis.
## OCamlPro ?
OCamlPro has been created to promote the use of OCaml in the industry. In order to do so, we provide a wide range of services targeted at all stages of typical software projects: we train engineers and we improve the efficiency and usability of the OCaml compiler and tools, we help design new projects, advise on which open-source software components to use and how, we help deliver OCaml software projects and we do custom software development. One extra focus is the increase of the accessibility of OCaml for beginners and students.
Our customers are well-known industrial users such as [Jane-Street](http://www.janestreet.com/), [Citrix](http://www.citrix.com/) and [Lexifi](http://www.lexifi.com/) but we also help individual developers lost in the wild of non-OCaml environments inter-operate OCaml with other components. We also believe that collaborative R&D projects are a great opportunity to make existing companies discover OCaml and its benefits to their products and we are involved in several of them (see below).
Our engineering team is steadily growing (currently 9 full-time engineers in a joint lab between OCamlPro and INRIA) located in Paris and Nice. We gather a wide range of technical skills and industrial world expertise as we are all coming from major academic and industrial actors such as [INRIA](http://www.inria.fr/), [text](Dassaut Systèmes), [MLstate](http://www.mlstate.com/) and [Citrix](http://www.citrix.com/). We also love the OCaml open-source ecosystem: we have been participating to the development of [ocsigen](http://ocsigen.org/), [mirage](http://www.openmirage.org/), [XCP](http://www.xen.org/products/cloudxen.html), [mldonkey](http://mldonkey.sourceforge.net/), [marionet](http://www.marionnet.org/EN/) and so on. By the way, OCamlPro has some open [positions](/jobs) and we are still looking to hire excellent software engineers!
## OCaml Distribution
The first of our technical activities is related to work on the OCaml distribution itself. We are part of the OCaml compiler development team - our INRIA members are part of the [Gallium](http://gallium.inria.fr/) project which develops OCaml at INRIA - and we regularly contribute patches to improve the usability and performance of the compiler itself.
We have recently proposed [a series of patches](http://caml.inria.fr/mantis/view.php?id=5894) to improve the performance of functions with float arguments and we have started developing a [framework](https://github.com/chambart/ocp-bench) to benchmark the efficiency of compiler optimizations.
We are also actively exploring the design-space for concurrency and distribution in OCaml, with an implementation of
- reentrant runtime
- way to instantiate different runtimes in separate system threads in the same process
- efficient multi-scale communication library, between threads and between processes.
We call this **multi-runtime OCaml** and a prototype is available on [github](https://github.com/lucasaiu/ocaml/tree/master).
Last, we are also making progress with the memory profiling tools. We work on a modified OCaml runtime which can store the location of each allocated block in the heap, with hooks to dump that heap on demand. External tools can then use that dump to produce useful statistics about memory usage of the program. The good news is that we now have a working and usable bytecode runtime and an external tool that produces basic memory information. We are preparing an alpha release in the next month, so stay tuned!
## Development Tools
Our efforts to make OCaml more usable go further than looking at the compiler. We are improving the development tools already existing in the community, such as the recently released [indentation tool](https://github.com/OCamlPro/ocp-indent) which was initially coming from an experiment from [Jun Furuse](https://bitbucket.org/camlspotter/ocaml-indent), and creating new ones when the lack is blatant.
Most recent news on that front concern [OPAM](http://opam.ocamlpro.com/), the package manager for OCaml that we are developing since mid-2012. For people not familiar with it yet, OPAM is a source-based package manager for OCaml. It supports resolution of complex dependency constraints between packages, multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development workflow. The beta release was announced in January, and we expect the first official release to happen in the next weeks. The OCaml community has gratefully welcomed OPAM, and the [repository of its package metadata](https://github.com/OCamlPro/opam-repository) has already become the most forked OCaml project on github! Interestingly, two meetups have gathered more than fifty OPAM users in [Paris](http://meetup.ocaml-lang.fr/) and Cambridge in January. We really hope this kind of meetup can be generalized: if you want to help us organize one in your area, feel free to contact us!
The other major part of our work around development tools for OCaml is TypeRex. TypeRex is a collection of tools which focus on improving developer efficiency, and lowering the entry barrier for experienced developers who are used to shiny IDEs in other languages. The first version of [TypeRex](http://www.typerex.org/), that was released last year, was a first step in this direction: it provided an enhanced emacs mode for OCaml code, with colorization, indentation, refactoring facilities, completion, code-navigation, documentation tooltips, etc. The next version of TypeRex (simply dubbed **typerex2**) is underway, with more independent tools (like ocp-indent), less tightly coupled to Emacs, and focused on better integration with various IDEs. If you are interested in following the progress of these tools, you can check the **typerex2** OPAM packages with 1.99.X+beta version numbers, which we release on a regular basis.
## R&D projects
The idea that OCaml is the right choice to create new innovative products is at the core of OCamlPro. We are very involved in the research community, especially on Functional Languages, with participation into the Program Committees of various conferences such as [the OCaml User and Developer (OUD)](http://oud.ocaml.org/) workshop and [the Commercial User of Functional Programming (CUFP)](http://cufp.org/) conference. We also joined two collaborative R&D projects in 2012, the [Richelieu FUI](http://www.richelieu.pro/) and [BWare ANR](http://bware.lri.fr/index.php/BWare_project). As part of the Richelieu project, we are developing a JIT compiler for the [Scilab language](http://www.scilab.org/). As part of the Bware project, we improve the efficiency of automatic theorem provers, with a specific focus on [Alt-Ergo](http://alt-ergo.lri.fr/), an SMT solver particularly suited to program verification. We are always interested in bringing our expertise in compiler technologies and knowledge of complex and distributed systems to new R&D projects: contact us if you are interested!
In the Richelieu project, our combined technical and theoretical expertise proved particularly effective. The research consortium is led by [Scilab Entreprises](http://www.scilab-enterprises.com/) which needed a safer and more efficient execution engine for Scilab in order to compete with Matlab. We joined the consortium to implement the early analysis required by the JIT compiler. The project started last December, and we have since specified the semantics of the language and implemented a working prototype of an interpreter that is already as fast as the current C++ engine of Scilab 6.
## Growing the Community
Our last important domain of activity is geared towards the OCaml community. It is important to us that the community grows bigger, and to achieve this goal there are some basic blocks that we need to help build, together with the other main actors of the community.
The first missing block is a good reference documentation. This year will end with (at least) one new important book for the language: [Real-World OCaml](http://www.realworldocaml.org/) which targets experienced software engineers who do not know OCaml yet. We collaborate with [OCamlLabs](http://www.cl.cam.ac.uk/projects/ocamllabs/) to make the technical experience of this book a success. We also work to improve the general experience of using OCaml for complete beginners by creating a stable [replacement](https://github.com/OCamlPro/ocp-edit-simple) to the broken **ocamlwin**, the simple editor distributed with OCaml on Windows.
It is also important to us that OCaml uses the web as a platform to attract new users, as is becoming the norm for modern programming languages. We are members of the [ocaml.org](http://www.ocaml.org/) building effort and have created [tryocaml](http://try.ocamlpro.com/) to let newcomers easily discover the language directly from their browser. TryOcaml has been welcomed as a great tool, already adopted and adapted: see for instance [tryrtt](http://rtt.forge.ocamlcore.org/tryrtt.html) or [try ReactiveML](http://rml.lri.fr/tryrml/). We are in the process of simplifying the integration with other compiler variants.
Last, but not least, we collaborate very closely with OCamlLabs to create the OCaml Plateform: a consistent set of libraries, thoroughly tested and integrated, with a rolling release schedule of 6 months. The platform will be based on [OPAM](http://opam.ocamlpro.com/) and we are currently designing and prototyping a testing infrastructure to improve and guarantee the quality of packages.
src/content/blog/2013_03_18_an_indentation_engine_for_ocaml.md 0 → 100644
View file @ b4e52ea0
title=An Indentation Engine for OCaml
authors=Louis Gesbert
date=2013-03-18
category=Tooling
tags=indentation,ocp-indent
Since our last [activity report](https://ocamlpro.com/2013/02/18/an-overview-of-our-current-activities/) we have released the first stable versions of two projects: [OPAM](https://opam.ocamlpro.com/), an installation manager for OCaml source packages, and [ocp-indent](https://github.com/OCamlPro/ocp-indent), an indentation tool.
We have already described the basics of OPAM in two precedent [blog](/blog/2013_01_17_beta_release_of_opam) [posts](/blog/2013_03_15_opam_1.0.0_released), so today we will focus on the release of `ocp-indent`.
### Indentation should be consistent across editors
When you work on a very large code-base, it is crucial to keep a
consistent indentation scheme. This is not only good for code review
purposes (when the indentation carries semantic properties) but also
when your code is starting to evolve and when the one who makes the
change is not the one who wrote the initial piece of code. In the latter
case, the variety of editors and local configurations usually leads to
lot of small changes carrying no semantic value at all (such as changing
tabs to spaces, adding few spaces at the beginning or end of lines, and
so on). This semantic noise considerably decreases the efficiency of
any code-review and change process and is usually very good at hiding
hard-to-track bugs in your code-base.
A few months ago, the solutions for OCaml to this indentation problem
were limited. For instance, you could write coding guidelines and hope
that all the developers in your project would follow them. If you wanted
to be more systematic, you could create and share a common
configuration file for some popular editors (most OCaml developers use
the emacs’ `tuareg-mode` or vim) but it is very hard to get
consistent indentation result across multiple tools. Moreover, having to
rely on a specific editor mode means that it is harder to fully
automatize the indentation process, for instance when setting-up a VCS
hook.
In order to overcome these pitfalls, [Jane Street](https://www.janestreet.com/) asked us to design a new external tool with the following high-level specification:
- it should be easy to use inside and outside any editor;
- it should understand the OCaml semantics and reflect it in the indentation;
- it should be easy to maintain and to extend;
So we started to look at the OCaml tools’ ecosystem and we found an early prototype of Jun Furuse’s [ocaml-indent](https://bitbucket.org/camlspotter).
The foundation looked great but the result on real-world code sources
was not as nice as it could be, so we decided to start from this base to
build our new tool, that we called `ocp-indent`. Today, `ocp-indent` and `ocaml-indent` do not have much code in common anymore, but the global architecture of the system remains the same.
### Writing an indentation engine for OCaml
An indentation engine may seem like a rather simple problem: given
any line in the program, we want to compute its indentation level, based
on the code structure.
It turns out to be much more difficult than that, mainly because
indentation is only marginally semantic, and, worse, is a matter of
taste and “proper layout”. In short, it’s not a problem that can be
expressed concisely, because one really does want lots of specific cases
handled “nicely”, depending on the on-screen layout — position of line
breaks — rather than the semantic structure. `Ocp-indent`
does contain lots of ad-hoc logic for such cases. To make things harder,
the OCaml syntax is known to be difficult to handle, with a few
ambiguities.
#### Indent process
`Ocp-indent` processes code in a simple and efficient way:
- We lex the input with a [modified version of the OCaml lexer](https://github.com/OCamlPro/ocp-indent/blob/master/src/approx_lexer.mll),
to guarantee complete consistency with OCaml itself. The parser had to
be modified to be more robust (ocaml fails on errors, the indentation
tool should not) and to keep tokens like comments, quotations, and, in
the latest version, some ocamldoc block delimiters.
- Taking the token stream as input, we maintain a [“block” stack](https://github.com/OCamlPro/ocp-indent/blob/master/src/indentBlock.ml)
that keeps informations like the kinds of blocks we have been through
to get to the cursor position, the column and the indentation
parameters. For instance, the “block” stack `[KBody KFfun; KLet; KBody KModule]` corresponds to the position of `X` in the following piece of (pseudo-) code:
```ocaml
…
module Foo = struct
…
let f = fun a &> X
```
- Each token may look up the stack to find its starting counterpart (`in` will look for `KLet`, etc.), or disambiguate (`=` will look for `KLet`, stopping on opening tokens like `KBracket`,
and will be inserted as an operator if none is found). This is flexible
enough to allow for “breaking” the stack when incorrect grammar is
found. For example, the unclosed paren in `module let x = ( end` should not break indent after the `end`. Great care was taken in deciding what tokens should be able to remove from the stack in which conditions.
- The stack can also be used to find a token that we want to align on, typically bars `|` in a pattern-matching.
- On every line break, the stack can be used to compute the indentation of the next line.
- In the case of partial file indentation (typically, reindenting one
line or a single block), on lines that shouldn’t be reindented the stack
is reversely updated to adapt to the current indentation.
#### Priorities
The part where some abstraction can be put into the engine is the
knowledge of the semantics, and more precisely of the scope of the
operations. It’s also in that case that the indenter can help you write,
and not only read, your code. On that matter, `ocp-indent`
has a knowledge of the precedence of operators and constructs that is
used to know how far to unwind the stack, and what to align on. For
example, a `;` will flush function applications and most operators.
It is that part that gives it the most edge over `tuareg`,
and avoids semantically incorrect indents. All infix operators are
defined with a priority, a kind of indentation (indentation increment or
alignment over the above concerned expression), and an indentation
value (number of spaces to add). So for example most operators have a
priority lower than function application, but not `.`, which yields correct results for:
```ocaml
let f =
somefun
record.
field
y
+ z
```
Boolean operators like `&&` and `||` are setup for alignment instead of indentation:
```ocaml
let r = a
|| b
&& c
|| d
```
Additionally, some special operators are wanted with a *negative* alignment in some cases. This is also handled in a generic way by the engine. In particular, this is the case for `;` or `|`:
```ocaml
type t = A
| B
let r = { f1 = x
; f2 = y
}
```
#### A note on the integration in editors
`ocp-indent` can be used on the command-line to reindent whole files (or part of them with `--lines`),
but the most common use of an indenter is from an editor. If you are
lucky enough to be able to call OCaml code from your editor, you can use
it directly as a library, but otherwise, the preferred way is to use
the option `--numeric`: instead of reprinting the file
reindented, it will only output indentation levels, which you can then
process from your editor (for instance, using `indent-line-to` with emacs). That should be cheaper and will help preserve cursor position, etc.
Currently, a simple emacs binding working on either the ocaml or the
tuareg mode is provided, together with a vim mode contributed by Raphaël
Proust and David Powers.
### Results
We’ve built `ocp-indent` based on a growing collection of [unit-tests](https://github.com/OCamlPro/ocp-indent/tree/master/tests/passing). If you find an indentation bug, feel free to [send us](https://github.com/OCamlPro/ocp-indent/issues) a code snippet that we will incorporate into our test suite.
Our tests clearly show that the deep understanding that `ocp-indent`
has of the OCaml syntax makes it shines on specific cases. We are still
discussing and evaluating the implementation of few corner-cases
related, see for instance the [currently failing tests](http://htmlpreview.github.com/?https://github.com/OCamlPro/ocp-indent/blob/master/tests/failing.html).
We have also run some [benchmarks](https://htmlpreview.github.com/?https://github.com/AltGr/ocp-indent-tests/blob/master/status.html) on real code-bases and the result is quite conclusive: `ocp-indent`
is always better than tuareg! This is a very nice result as most of the
existing source files are either indented manually or are following
tuareg standards. But `ocp-indent` is also orders of magnitude faster, which means you can integrate it seamlessly into any automatic process.
src/content/blog/2013_04_22_april_monthly_report.md 0 → 100644
View file @ b4e52ea0
title=April Monthly Report
authors=Çagdas Bozman
date=2013-04-22
category=OCamlPro
tags=Memory,OCaml,OCamlPro,ocp-indent,Profiling,Tools
This post aims at summarizing the activities of OCamlPro for the past month. As usual, we worked in three main areas: the OCaml toolchain, development tools for OCaml and R&D projects.
## The toolchain
Our multi-runtime implementation of OCaml had gained stability. [Luca](http://ageinghacker.net/) fixed a lot of low-level bugs in the “master” branch of [his OCaml repository](http://www.github.com/lucasaiu/ocaml), which were mainly related to the handling of signals. There are still some issues, which seem to be related to thread-switching (ie. when using OS level mutli-threading).
We made great progress on improved inlining strategy. In the current OCaml compiler, inlining, closure conversion and constant propagation are done in a single pass interleaved with analysis. It has served well until now, but to improve it in a way which is easily extensible in the future, it needs a complete rewrite. After a few prototypes, [Pierre](http://www.lsv.ens-cachan.fr/%7Echambart/) is now coming up with a suitable intermediate language (IR) more suited for the job, using a dedicated value analysis to guide the simplification and inlining passes. This IR will stand between the lambda code and the C-lambda and is designed such that future specialized optimization can be easily be added. There are two good reasons for this IR: First, it is not as intrusive and reduces the extent of the modifications to the compiler, as it can be plugged between two existing passes and turned on or off using a command-line flag. Second, it can be tweaked to make the abstract interpretation more precise and efficient. For instance, we want the inlining to work with higher-order functions as well as modules and functors, performing basic defunctorization. It is still in an experimentation phase, but we are quickly converging on the API and hope to have something we can demo in the next months.
Our [frame-pointer patch](http://ocamlpro.com/2012/08/08/profiling-ocaml-amd64-code-under-linux/) has also been accepted. Note that, as this patch changes the calling sconvention of OCaml programs, you cannot link together libraries compiled with and without the patch. Hence, this option will be provided as a configuration switch (`./configure --with-frame-pointer`).
Regarding memory profiling, we released a preliminary prototype of the memory profiler for native code. It is available in [Çagdas](https://github.com/cago/ocaml) repository. We are still in the process of testing and evaluating the prototype before making it widely available through OPAM. As the previous bytecode prototype, you need to compile the libraries and the program you want to profile as usual in order to build a table associating unique identifier to program locations (.prof file). Then, for each allocated block, we have then patched the runtime of OCaml to encode in the header the identifier of the line which allocated it. To be able to dump the heap, you can either instrument your program, or send a signal, or set the appropriate environment variable (`OCAMLRUNPARAM=m`). Finally, you can use the profiler which will read the .prof and .cmt files in order to generate a pdf file which is the amount of memory use by type. More details on this will come soon, you can already read the [README](https://github.com/cago/ocaml/blob/4.00.1%2Bmemprof/README) file available on github.
Finally, we organized a new meeting with the core-team to discuss some of the bugs in the [OCaml bug tracker](http://caml.inria.fr/mantis). It was the first of the year, but we are now going to have one every month, as it has a very positive impact on the involvement of everybody in fixing bugs and helps focus work on the most important issues.
## Development Tools for OCaml
Since the latest release of [ocp-indent](http://github.com/OCamlPro/ocp-indent), [Louis](http://louis.gesbert.fr/cv.en.html) continued to improve the tool. We plan to release version 1.2.0 in the next couple of days, with some bug fixes (esp. related to the handling of records) and the following new features: operators are now aligned by default (as well as opened parentheses not finishing a line) and indentation can be bounded using the `max_indent` parameter. We are also using the great [cmdliner](http://erratique.ch/software/cmdliner) which means `ocp-indent` now has nice manual pages.
We are also preparing a new minor release of [OPAM](http://opam.ocamlpro.com/), with a few bug fixes, an improved solver heuristic and improved performance. OPAM statistics seem to converge towards round numbers, as [OcamlPro/opam](http://github.com/OCamlPro/opam) repository has recently reached 100 “stars” on Github, [OCamlPro/opam-repository](http://github.com/OCamlPro/opam-repository) is not very far from being forked 100 times, while the number of unique packages on [opam.ocamlpro.com](http://opam.ocamlpro.com) is almost 400\. We are also preparing the platform release, with a cleaner and simpler client API to be used by the upcoming “Ocamlot”, the automated infrastructure which will test and improve the quality and consistency of OPAM packages.
Last, we released a very small – but already incredibly useful tool: [ocp-index](http://github.com/OCamlPro/ocp-index). This new tool provides completion based on what is installed on your system, with type and documentation when available. Similarly to `ocp-indent`, the main goal of this tool is to make it easy to integrate in your editor of choice. As a proof of concept, we also distribute a small curses-based tool, called `ocp-browser`, which lets you browse interactively the libraries installed on your system, as well as an emacs binding for `auto-complete.el`. Interestingly enough, behind the scene `ocp-index` uses a [lazy immutable prefix tree](https://github.com/OCamlPro/ocp-index/blob/master/src/trie.mli) with merge operations to efficiently store and load cmis and cmt files.
## Other R&D Projects
We continued to work on the [Richelieu](http://www.richelieu.pro/) project. We are currently adding basic type-inference for Scilab programs to our tool [scilint](https://github.com/OCamlPro/richelieu/tree/jit-fabrice/scilab/modules/jit_ocaml/src/scilint), to be able to print warnings on possible programers mistakes. A first part of the work was to understand how to automatically get rid of some of the `eval` constructs, especially `deff` and `evalstr` primitives that are often used. After this, [Michael](https://github.com/Michaaell) manually analyzed some real-world Scilab programs to understand how typing should be done, and he is now implementing the type checker and a table of types for primitive functions.
We are also submitting a new project, called SOCaml, for funding by the French government. In 2010, [ANSSI](http://www.ssi.gouv.fr/), the French agency for the security of computer systems, commanded a study, called LAFOSEC, to understand the advantages of using functional languages in the domain of security. Early results of the study were presented in [JFLA’2013](http://jfla.inria.fr/2013/programme), with in particular recommandations on how to improve OCaml to use it for security applications. The goal of the SOCaml project would be to implement these recommandations, to improve OCaml, to provide additional tools to detect potential errors and to implement libraries to verify marshaled values and bytecode. We hope the project will be accepted, as it opens a new application domain for OCaml, and would allow us to work on this topic with our partners in the project, such as [LexiFi](http://www.lexifi.com) and [Michel Mauny](http://michel.mauny.net/)‘s team at ENSTA Paristech (the project would also contribute to their [ocamlcc](http://github.com/ocaml-bytes/ocamlcc) bytecode-to-c compiler).
\ No newline at end of file
src/content/blog/2013_07_11_better_inlining_progress_report.md 0 → 100644
View file @ b4e52ea0
title=Better Inlining: Progress Report
authors=chambart
date=2013-07-11
category=OCaml
tags=OCaml,inlining,flambda-prequel
As announced [some time ago](optimisations-you-shouldnt-do), I am working on a new intermediate language within the OCaml compiler to improve its inlining strategy. After some time of bug squashing, I prepared a testable version of the patchset, available either on [Github](https://github.com/chambart/ocaml.git) (branch `flambda_experiments`), or through OPAM, in the following repository:
```shell-session
opam repo add inlining https://github.com/OCamlPro/opam-compilers-repository.git
opam switch flambda
opam install inlining-benchs
```
The series of patches is not ready for benchmarking against real applications, as no cross module information is propagated yet (this is more practical for development because it simplifies debugging a lot), but it already works quite well on single-file code. Some very simple benchmark examples are available in the `inlining-benchs` package.
The series of patches implements a set of 'reasonable' compilation passes, that do not try anything too complicated, but combined, generates quite efficient code.
## Current Status
As said in the previous post, I decided to design a new intermediate language to implement better inlining heuristics in the compiler. This intermediate language, called `flambda`, lies between the `lambda` code and the `Clambda` code. It has an explicit representation of closures, making them easier to manipulate, and modules do not appear in it anymore (they have already been compiled to static structures).
I then started to implement new inlining heuristics as functions from the `lambda` code to the `flambda` code. The following features are already present:
* intra function value analysis
* variable rebinding
* dead code elimination (which needs purity analysis)
* known match / if branch elimination
In more detail, the chosen strategy is divided into two passes, which can be described by the following pseudo-code:
```
if function is at toplevel
then if applied to at least one constant OR small enough
then inline
else if applied to at least one constant AND small enough
then inline
```
```
if function is small enough
AND does not contain local function declarations
then inline
```
The first pass eliminates most functor applications and functions of the kind:
```ocaml
let iter f x =
let rec aux x = ... f ... in
aux x
```
The second pass eliminates the same kind of functions as Ocaml 4.01, but being after the first pass, it can also inline functions revealed by inlining functors.
## Benchmarks
I ran a few benchmarks to ensure that there were no obvious miscompilations (and there were, but they are now fixed). On benchmarks that were too carefully written there was not much gain, but I got interesting results on some examples: those illustrate quite well the improvements, and can be seen at `$(opam config var lib)/inlining-benchs` (binaries at `$(opam congfig var bin)/bench-*`).
### The Knuth-Bendix Benchmark (single-file)
Performance gains against OCaml 4.01 are around 20%. The main difference is that exceptions are compiled to constants, hence not allocated when raised. In that particular example, this halves the allocations.
In general, constant exceptions can be compiled to constants when predefined (`Not_found`, `Failure`, ...). They cannot yet when user defined: to improve this a few things need to be changed in `translcore.ml` to annotate values created by exceptions.
### The Noiz Benchmark:
Performance gains are around 30% against OCaml 4.01. This code uses a lot of higher order functions of the kind:
```ocaml
let map_triple f (a,b,c) = (f a, f b, f c)
```
OCaml 4.01 can inline `map_triple` itself but then cannot inline the parameter `f`. Moreover, when writing:
```ocaml
let (x,y,z) = map_triple f (1,2,3)
```
the tuples are not really used, and after inlining their allocations can be eliminated (thanks to rebinding and dead code elimination)
### The Set Example
Performance gains are around 20% compared to OCaml 4.01. This example shows how inlining can help defunctorization: when inlining the `Set` functor, the provided comparison function can be inlined in `Set.add`, allowing direct calls everywhere.
## Known Bugs
### Recursive Values
A problem may arise in a rare case of recursive values where a field access can be considered to be a constant. Something that would look like (if it were allowed):
```ocaml
type 'a v = { v : 'a }
let rec a = { v = b }
and b = (a.v, a.v)
```
I have a few solutions, but not sure yet which one is best. This probably won't appear in any normal test. This bug manifests through a segmentation fault (`cmmgen` fails to compile that recursive value reasonably).
### Pattern-Matching
The new passes assume that every identifier is declared only once in a given module, but this assumption can be broken on some rare pattern matching cases. I will have to dig through `matching.ml` to add a substitution in these cases. (the only non hand-built occurence that I found is in `ocamlnet`)
## Known Mis-compilations
* since there is no cross-module information at the moment, calls to functions from other modules are always slow.
* In some rare cases, there could be functions with more values in their closure, thus resulting in more allocations.
## What's next ?
I would now like to add back cross-module information, and after a bit of cleanup the first series of patches should be ready to propose upstream.
src/content/blog/2013_09_04_ocamlpro_highlights_august_2013.md 0 → 100644
View file @ b4e52ea0
title=OCamlPro Highlights, August 2013
authors=Çagdas Bozman
date=2013-09-04
category=OCamlPro
tags=highlights
Here is a short report on the different projects we have been working on in August.
### News from OCamlPro
#### Compiler Optimizations
After our reports on [better inlining](https://ocamlpro.com/2013/07/11/better-inlining-progress-report/)
have raised big expectations, we have been working hard on fixing the
few remaining bugs. An enhanced alias/constant analysis was added, to
provide the information needed to lift some constraints on the
maintained invariants, and simplifying some other passes quite a lot in
the process. We are now working on reestablishing cross-module inlining,
by exporting the new information between compilation units.
#### Memory Profiling
On the memory profiling front, now that the compiler patch is well
tested and quite stable, we started some cleanup to make it more
modular, easier to understand and extend. We also worked on improving
the performance of the profiler (the tool that analyzes the heap
snapshots), by caching some expensive computations, such as extracting
type information from ‘cmt’ files associated with each location, in
files that are shared between executions. We have started testing the
profiler on [the Why3 verification platform](https://why3.lri.fr/), and these optimizations proved very useful to analyze longer traces.
#### OPAM Package Manager
On OPAM, we are still preparing the release of version 1.1. The
release date has shifted a little bit — it is now planned to happen
mid-September, before the [OCaml’2013 meeting](https://ocaml.org/meetings/ocaml/2013/) — because we are focusing on getting speed and stability improvements in a very good shape. We are now relying on [opam-rt](https://github.com/OCamlPro/opam-rt), our new regression testing infrastructure, to be sure to get the best level of quality for the release.
Regarding the package and compiler [metadata](https://github.com/OCamlPro/opam-repository),
we are very proud to announce that our community has crossed an
important line, with more than 100 contributors and 500 different
packages ! In order to ensure that these hours of packaging efforts
continue to benefit everyone in the OCaml community in the future, we
are (i) clarifying the license for all the metadata in the package
repository to use [CC0](https://github.com/OCamlPro/opam-repository/issues/955) and (ii) discussing with [OCamlLabs](https://www.cl.cam.ac.uk/projects/ocamllabs/) and the different stakeholders to migrate all the metadata to the [ocaml.org](https://ocaml.org/) infrastructure.
#### Simple Build Manager
We also made progress on the design of our simple build-manager for OCaml, [ocp-build](https://www.typerex.org/ocp-build.html). The [next branch in the GIT repository](https://github.com/OCamlPro/ocp-build/tree/next)
features a new, much more expressive package description language :
ocp-build can now be used to build arbitrary files, for example to
generate new source files, or to compile files in other languages than
OCaml. We successfully used the new language to build [Try-ocaml](https://try.ocamlpro.com/) and [wxOCaml](https://www.typerex.org/ocplib-wxOCaml.html), completely avoiding the use of “make”.
It can also automatically generate [basic HTML documentation](https://www.typerex.org/ocplib-wxOCaml/doc.0.1/index.html)
for libraries using ocamldoc with “ocp-build make -doc”. There are
still some improvements on our TODO list before an official release, for
example improving the support of META files, but we are getting very
close ! ocp-build is very efficient: compiling [Merlin with ocp-build](https://www.typerex.org/ocp-build/merlin.ocp) takes only 4s on a quad-core while ocamlbuild needs 13s in similar conditions and with the same parallelisation settings.
#### Graphics on Try-OCaml
[Try-OCaml](https://try.ocamlpro.com/) has been improved
with a dedicated implementation of the Graphics module: type “lesson
19”, and you will get some fun examples, including a simple game by
Sylvain Conchon.
#### Alt-Ergo Theorem Prover
We are also happy to welcome Mohamed Iguernelala in the team,
starting at the beginning of September. Mohamed is a great OCaml
programmer, and he will be working on the Alt-Ergo theorem prover, an
SMT-solver in OCaml developed by Sylvain Conchon, and heavily used in
the industry for safety-critical software (aircrafts, trains, etc.).