Commit f5eac3f4 authored by Dario Pinto's avatar Dario Pinto
Browse files

Merge branch 'z-2021-10-05-blog-articles' into 'master'

Add Fabrice's articles

See merge request OCamlPro/www!42
parents 74dcbc4e d0b019bc
......@@ -7,17 +7,17 @@ tags=OCamlPro,String,OCaml
You will need OCaml 3.11.2 installed on a i686 linux computer. The archive contains:
- libcamlrun-linux-i686.a
- ocamlrun-linux-i686
- Makefile
- README
* libcamlrun-linux-i686.a
* ocamlrun-linux-i686
* Makefile
* README
The Makefile has two targets:
- **sudo make install** will save */usr/bin/ocamlrun* and */usr/lib/ocaml/libcamlrun.a* in the current directory and replace them with the longval binaries.
- **sudo make restore** will restore the saved files.
- `sudo make install` will save `/usr/bin/ocamlrun` and `/usr/lib/ocaml/libcamlrun.a` in the current directory and replace them with the longval binaries.
- `sudo make restore` will restore the saved files.
If your install directories are not the default ones, you should modify the Makefile. After installing, you can test it with the standard OCaml top-level:
......@@ -25,13 +25,18 @@ If your install directories are not the default ones, you should modify the Make
`Objective Caml version 3.11.2`
```Ocaml
# let s = ref “”;;
val s : string ref = {contents = “”}
# s := String.create 20_000_000;;
– : unit = ()
```
Now you can enjoy big values in all your strings and arrays in bytecode. You will need to relink all your custom binaries. If you are interested in the native version of the longval compiler, you can [contact](mailto:contact@ocamlpro.com) us.
Now you can enjoy big values in all your strings and arrays in
bytecode. You will need to relink all your custom binaries. If you are
interested in the native version of the longval compiler, you can
[contact](mailto:contact@ocamlpro.com) us.
......
title=Packing and Functors
authors=Fabrice Le Fessant
date=2011-08-10
category=OCaml
tags=ocaml,tooling
We have recently worked on modifying the OCaml system to be able to
pack a set of modules within a functor, parameterized on some
signatures. This page presents this work, funded by Jane Street.
All the patches on this page are provided for OCaml version 3.12.1.
## Packing Functors
### Installation of the modified OCaml system
The patch for OCaml 3.12.1 is available here:
```
ocaml+libfunctor-3.12.1.patch.gz (26 kB)
```
To use it, you can use the following recipe, that will compile and
install the patched version in `~/ocaml+libfunctor-3.12.1/bin/`.
```shell-session
~% wget http://caml.inria.fr/pub/distrib/ocaml-3.12/ocaml-3.12.1.tar.gz
~% tar zxf ~/ocaml-3.12.1.tar.gz
~% cd ocaml-3.12.1
~/ocaml-3.12.1% wget ocamlpro.com/code/ocaml+libfunctor-3.12.1.patch.gz
~/ocaml-3.12.1% gzip -d ocaml+libfunctor-3.12.1.patch.gz
~/ocaml-3.12.1% patch -p1 < ocaml+libfunctor-3.12.1.patch
~/ocaml-3.12.1% ./configure –prefix ~/ocaml+libfunctor-3.12.1
~/ocaml-3.12.1% make coldstart
~/ocaml-3.12.1% make ocamlc ocamllex ocamltools
~/ocaml-3.12.1% make library-cross
~/ocaml-3.12.1% make bootstrap
~/ocaml-3.12.1% make all opt opt.opt
~/ocaml-3.12.1% make install
~/ocaml-3.12.1% cd ~
~% export PATH=$HOME/ocaml+libfunctor-3.12.1/bin:$PATH
```
Note that it needs to bootstrap the compiler, as the format of object
files is not compatible with the one of ocaml-3.12.1.
### Usage of the lib-functor patch.
Now that you are equiped with the new system, you can start using it. The lib-functor patch adds two new options to the compilers ocamlc and ocamlopt:
* `-functor <interface_file>` : this option is used to specify that the current module is compiled with the interface files specifying the argument of the functor. This option should be used together with -for-pack <module>, where <module> is the name of the module in which the current module will be embedded.
* `-pack-functor <module>` : this option is used to pack the modules. It should be used with the option -o <object_file> to specify in which module it should be embedded. The <module> specified with -pack-functor specifies the name of functor that will be created in the target object file.
If the interface x.mli contains :
```ocaml
type t
val compare : t -> t -> int
```
and the files `xset.ml` and `xmap.ml` contain respectively :
```ocaml
module T = Set.Make(X)
```
```ocaml
module T = Map.Make(X)
```
Then :
```shell-session
~/test% ocamlopt -c -for-pack Xx -functor x.cmi xset.ml
~/test% ocamlopt -c -for-pack Xx -functor x.cmi xmap.ml
~/test% ocamlopt -pack-functor MakeSetAndMap -o xx.cmx xset.cmx xmap.cmx
```
will construct a compiled unit whose signature is (that you can get
with `ocamlopt -i xx.cmi`, see below) :
```ocaml
module MakeSetAndMap :
functor (X : sig type t val compare : t -> t -> int end) -> sig
module Xset : sig
module T : sig
type elt = X.t
type t = Set.Make(X).t
val empty : t
val is_empty : t -> bool
end
end
module Xmap : sig
module T : sig
type key = X.t
type a t = a Map.Make(X).t
val empty : a t
val is_empty : a t -> bool
end
end
end
```
### Other extension: printing interfaces
OCaml only allows you to print the interface of a module or interface
by compiling its source with the -i option. However, you don’t always
have the source of an object interface (in particular, if it was
generated by packing), and you might still want to do it.
In such a case, the lib-functor patch allows you to do that, by using
the -i option on an interface object file:
```shell-session
~/test% cat > a.mli
val x : int
~/test% ocamlc -c -i a.mli
val x : int
~/test% ocamlc -c -i a.cmi
val x : int
```
### Other extension: packing interfaces
OCaml only allows you to pack object files inside another object file
(.cmo or .cmx). When doing so, you can either provide an source
interface (.mli) that you need to compile to provide the corresponding
object interface (.cmi), or the object interface will be automatically
generated by exporting all the sub-modules within the packed module.
However, sometimes, you would want to be able to specify the
interfaces of each module separately, so that:
* you can reuse most of the interfaces you already specified
* you can use a different interface for a module, that the one used to
compile the other modules. This happens when you want to export more
values to the other internal sub-modules than you want to export to
the user.
In such a case, the lib-functor patch allows you to do that, by using
the -pack option on interface object files:
```shell-session
test% cat > a.mli
val x : int
test% cat > b.mli
val y : string
test% ocamlc -c a.mli b.mli
test% ocamlc -pack -o c.cmi a.cmi b.cmi
test% ocamlc -i c.cmi
module A : sig val x : int end
module B : sig val y : string end
```
## Using `ocp-pack` to pack source files
### Installation of ocp-pack
Download the source file from:
`ocp-pack-1.0.1.tar.gz` (20 kB, GPL Licence, Copyright OCamlPro SAS)
Then, you just need to compile it with:
```shell-session
~% tar zxf ocp-pack-1.0.1.tar.gz
~% cd ocp-pack-1.0.1
~/ocp-pack-1.0.1% make
~/ocp-pack-1.0.1% make install
```
### Usage of `ocp-pack`
`ocp-pack` can be used to pack source files of modules within just one
source file. It allows you to avoid the use of the `-pack` option, that
is not always supported by all ocaml tools (for example,
`ocamldoc`). Moreover, `ocp-pack` tries to provide the correct locations
to the compiler, so errors are not reported within the generated
source file, but within the original source files.
It supports the following options:
```shell-session
% ocp-pack -help
Usage:
ocp-pack -o target.ml [options] files.ml*
Options:
-o <filename.ml> generate filename filename.ml
-rec use recursive modules
all .ml files must have a corresponding .mli file
-pack-functor <modname> create functor with name <modname>
-functor <filename.mli> use filename as an argument for functor
-mli output the .mli file too
.ml files without .mli file will not export any value
-no-ml do not output the .ml file
-with-ns use directory structure to create a hierarchy of modules
-v increment verbosity
–version display version information
```
`ocp-pack` automatically detects interface sources and implementation
sources. When only the interface source is available, it is assumed
that it is a type-only module, i.e. no val items are present inside.
Here is an example of using `ocp-pack` to build the ocamlgraph package:
```shell-session
test% ocp-pack -o graph.ml \
lib/bitv.ml lib/heap.ml lib/unionfind.ml \
src/sig.mli src/dot_ast.mli src/sig_pack.mli \
src/version.ml src/util.ml src/blocks.ml \
src/persistent.ml src/imperative.ml src/delaunay.ml \
src/builder.ml src/classic.ml src/rand.ml src/oper.ml \
src/path.ml src/traverse.ml src/coloring.ml src/topological.ml \
src/components.ml src/kruskal.ml src/flow.ml src/graphviz.ml \
src/gml.ml src/dot_parser.ml src/dot_lexer.ml src/dot.ml \
src/pack.ml src/gmap.ml src/minsep.ml src/cliquetree.ml \
src/mcs_m.ml src/md.ml src/strat.ml
test% ocamlc -c graph.ml
test% ocamlopt -c graph.ml
```
The -with-ns option can be used to automatically build a hierarchy of
modules. With that option, sub-directories are seen as
sub-modules. For example, packing a/x.ml, a/y.ml and b/z.ml will give
a result like:
[code language=”fsharp”]
module A = struct
module X = struct … end
module Y = struct … end
end
module B = struct
module Z = struct … end
end
[/code]
### Packing modules as functors
The `-pack-functor` and `-functor` options provide the same behavior
as the same options with the lib-functor patch. The only difference is
that `-functor` takes the interface source as argument, not the
interface object.
### Packing recursive modules
When trying to pack modules with `ocp-pack`, you might discover that
your toplevel modules have recursive dependencies. This is usually
achieved by types declared abstract in the interfaces, but depending
on each other in the implementations. Such modules cannot simply
packed by `ocp-pack`.
To handle them, `ocp-pack` provides a `-rec` option. With that option,
modules are put within a module rec construct, and are all required to
be accompagnied by an interface source file.
Moreover, in many cases, OCaml is not able to compile such recursive modules:
* For typing reasons: recursive modules are typed in an environment
containing only an approximation of other recursive modules
signatures
* For code generation reasons: recursive modules can be reordered
depending on their shape, and this reordering can generate an order
that is actually not safe, leading to an exception at runtime
To solve these two issues in most cases, you can use the following
patch (you can apply it using the same recipe as for lib-functor, and
even apply both patches on the same sources):
* `ocaml+rec-3.12.1.patch.gz`
With this patch, recursive modules are typed in an environment that is
enriched progressively with the final types of the modules as soon as
they become available. Also, during code generation, a topological
order is computed on the recursive modules, and the subset of modules
that can be initialized using in that topological order are immediatly
generated, leaving only the other modules to be reordered.
......@@ -6,7 +6,7 @@ tags=OCaml,inlining,flambda-prequel
As announced [some time ago](optimisations-you-shouldnt-do), I am working on a new intermediate language within the OCaml compiler to improve its inlining strategy. After some time of bug squashing, I prepared a testable version of the patchset, available either on [Github](https://github.com/chambart/ocaml.git) (branch `flambda_experiments`), or through OPAM, in the following repository:
```
```shell-session
opam repo add inlining https://github.com/OCamlPro/opam-compilers-repository.git
opam switch flambda
opam install inlining-benchs
......
title=OCamlPro’s Contributions to OCaml 4.00.0
authors=Fabrice Le Fessant
date=2012-08-20
category=OCaml
tags=ocaml,tooling
OCaml 4.00.0 has been released on July 27, 2012. For the first time,
the new OCaml includes some of the work we have been doing during the
last year. In this article, I will present our main contributions,
mostly funded by Jane Street and Lexifi.
## Binary Annotations for Advanced Development Tools
OCaml 4.00.0 has a new option `-bin-annot` (undocumented, for now, as
it is still being tested). This option tells the compiler to dump in
binary format a compressed version of the typed tree (an abstract
syntax tree with type annotations) to a file (with the `.cmt`
extension for implementation files, and `.cmti` for interface
files). This file can then be used by development tools to provide new
features, based on the full knowledge of types in the sources. One of
the first tools to use it is the new version of `ocamlspotter`, by Jun
Furuse.
This new option will probably make the old option `-annot` obsolete
(except, maybe, in specific contextes where you don’t want to depend
on the internal representation of the typedtree, for example when you
are modifying this representation !). Generated files are much smaller
than with the `-annot` option, and much faster to write (during
compilation) and to read (for analysis).
## New Options for ocamldep
As requested on the bug tracker, we implemented a set of new options for ocamldep:
* `-all` will print all the dependencies, i.e. not only on .cmi, .cmo and .cmx files, but also on source files, and for .o files. In this mode also, no proxying is performed: if there is no interface file, a bytecode dependency will still appear against the .cmi file, and not against the .cmo file as it would before;
* `-one-line` will not break dependencies on several lines;
* `-sort` will print the arguments of ocamldep (filenames) in the order of dependencies, so that the following command should work when all source files are in the same directory:
```shell-session
ocamlopt -o my_program `ocamldep -sort *.ml *.mli
```
## CFI Directives for Debugging
OCaml tries to make the best use of available registers and stack
space, and consequently, its layout on the stack is much different
from the one of C functions. Also, function names are mangled to make
them local to their module. As a consequence, debugging native code
OCaml programs has long been a problem with previous versions of
OCaml:, since the debugger cannot print correctly the backtrace of the
stack, nor put breakpoints on OCaml functions.
In OCaml 4.00.0, we worked on a patch submitted on the bug tracker to
improve the situation: x86 and amd64 backends now emit more debugging
directives, such as the locations in the source corresponding to
functions in the assembly (so that you can put breakpoints at function
entry), and CFI directives, indicating the correct stack layout, for
the debugger to correctly unwind the stack. These directives are part
of the DWARF debugging standard.
Unfortunately, line by line stepping is not yet available, but here is an example of session that was not possible with previous versions:
```ocaml
let f x y = List.map ( (+) x ) y
let _ = f 3 [1;2;3;4]
```
```shell-session
$ ocamlopt -g toto.ml
$ gdb ./a.out
(gdb) b toto.ml:1
Breakpoint 1 at 0x4044f4: file toto.ml, line 1.
(gdb) run
Starting program: /home/lefessan/ocaml-4.00.0-example/a.out
Breakpoint 1, 0x00000000004044f4 in camlToto__f_1008 () at toto.ml:1
1 let f x y = List.map ( (+) x ) y
(gdb) bt
0 0x00000000004044f4 in camlToto__f_1008 () at toto.ml:1
1 0x000000000040456c in camlToto__entry () at toto.ml:2
2 0x000000000040407d in caml_program ()
3 0x0000000000415fe6 in caml_start_program ()
4 0x00000000004164b5 in caml_main (argv=0x7fffffffe3f0) at startup.c:189
5 0x0000000000408cdc in main (argc=<optimized out>, argv=<optimized out>)
at main.c:56
(gdb)
```
## Optimisation of Partial Function Applications
Few people know that partial applications with multiple arguments are
not very efficient. For example, do you know how many closures are
dynamically allocated in in the following example ?
```ocaml
let f x y z = x + y + z
let sum_list_offsets orig list = List.fold_left (f orig) 0 list
let sum = sum_list_offsets 10 [1;2;3]
```
Most programmers would reply one, `f orig`, but that’s not all
(indeed, f and sum_list_offsets are allocated statically, not
dynamically, as they have no free variables). Actually, three more
closures are allocated, when `List.fold_left` is executed on the list,
one closure per element of the list.
The reason for this is that Ocaml has only two modes to execute
functions: either all arguments are present, or just one
argument. Prior to 4.00.0, when a function would enter the second mode
(as f in the previous example), then it would remain in that mode,
meaning that the two other arguments would be passed one by one,
creating a partial closure between them.
In 4.00.0, we implemented a simple optimization, so that whenever all
the remaining expected arguments are passed at once, no partial
closure is created and the function is immediatly called with all its
arguments, leading to only one dynamic closure creation in the
example.
## Optimized Pipe Operators
It is sometimes convenient to use the pipe notation in OCaml programs, for example:
```ocaml
let (|>) x f = f x;;
let (@@) f x = f x;;
[1;2;3] |> List.map (fun x -> x + 2) |> List.map print_int;;
List.map print_int @@ List.map (fun x -> x + 1 ) @@ [1;2;3];;
```
However, such `|>` and `@@` operators are currently not optimized: for
example, the last line will be compiled as:
```ocaml
let f1 = List.map print_int;;
let f2 = List.map (fun x -> x + 1);;
let x = f2 [1;2;3;];;
f1 x;;
```
Which means that partial closures are allocated every time a function
is executed with multiple arguments.
In OCaml 4.00.0, we optimized these operators by providing native
operators, for which no partial closures are generated:
```ocaml
external (|>) : a -> (a -> b) -> b = "%revapply";;
external ( @@ ) : (a -> b) -> a -> b = "%apply"
```
Now, the previous example is equivalent to:
```ocaml
List.map print_int (List.map ( (+) 1 ) [1;2;3])
```
## Bug Fixing
Of course, a lot of our contributions are not always as visible as the
previous ones. We also spent a lot of time fixing small bugs. Although
it doesn’t sound very fun, fixing bugs in OCaml is also fun, because
bugs are often challenging to understand, and even more challenging to
remove without introducing new ones !
title=OCamlPro Highlights: November 2013
authors=Fabrice Le Fessant
date=2013-12-03
category=OCaml
tags=ocaml,tooling
## New Team Members
We are pleased to welcome three new members in our OCamlPro team since the beginning of November:
* Benjamin Canou started working at OCamlPro on the Richelieu project,
an effort to bring better safety and performance to the
[Scilab](https://www.scilab.org/) language. He is in charge of a
type inference algorithm that will serve both as a developper tool
and in coordination with a JIT. He spent his first month
understanding the darkest corners of the language, and then writing
a versatile AST with a parser to build it. Actually, this is not an
easy task, because the language gives different statuses to
characters (including spaces) depending on the context, leading to
non-trivial lexing. But the real source of problems is the fact that
the original lexparser is intermingled with the interpreter inside a
big bunch of venerable FORTRAN code. This old fellow makes parsing
choices depending on the dynamic typing context, allows its users to
catch syntax errors at runtime, among other fun things. The new
OCaml lexer and parser is handwritten in around a thousand lines,
has performance comparable to a `Lex` and `Yacc` generated one, and
is resilient to errors so it could be integrated into an IDE to
detect errors on the fly without stopping on the first one. Once
again, it’s OCaml to the rescue of the weak and elderly!An example
of the kind of code that can be written in Scilab:
```scilab
if return = while then [ 12..
34.. … .. …
56 } ; else ‘”‘”
end
```
which is parsed into:
```scilab
— parsed in 0.000189–
(script (if (== !return !while) (matrix (row 123456)) “‘”))
— messages
1.10:1.11: use of deprecated operator ‘=’
— end
```
* Gregoire Henry started working at OCamlPro on the Bware project. He
is tackling the optimization of memory performance of automatic
provers written in OCaml, in collaboration with Cagdas Bozman. One
of his first contributions after joining us was to exhume his
internship work of 2004, an implementation of [Graphics for Mac OS
X](https://github.com/OCamlPro/ocplib-graphics) that we are going to
use for our online OCaml IDE!
* Thomas Blanc started a PhD at OCamlPro after his summer internship
with us. He is going to continue his work on whole-program analysis,
especially as a way to detect uncaught exceptions. We hope his tool
will be a good replacement for the [ocamlexn
tool](https://github.com/OCamlPro/ocamlexc)
written by Francois Pessaux.
## Compiler Updates
On the compiler optimization front, Pierre Chambart got direct access
to the OCaml SVN, so that he will soon upload his work directly into
an SVN branch, easier for reviewing and integration into the official
compiler. A current set of optimizations is already scheduled for the
new branch, and is now working on inlining recursive functions, such
`List.map`, by inlining the function definition at the call site, when
at least one of its arguments is invariant during recursion.
A function that can benefit a lot from that transformation is:
```ocaml
let f l = List.fold_left (+) 0 l
```
```
camlTest__f_1013:
.L102:
movq %rax, %rdi
movq $1, %rbx
jmp camlTest__fold_left_1017@PLT
camlTest__fold_left_1017:
.L101:
cmpq $1, %rdi
je .L100
movq 8(%rdi), %rsi
movq (%rdi), %rdi
leaq -1(%rbx, %rdi), %rbx
movq %rsi, %rdi
jmp .L101
.align 4
.L100:
movq %rbx, %rax
ret
```