Building Lingua::LinkParser from source was not as easy as I expected, but we got there in the end (kinda).
Note: If you want the tl;dr go to the bottom of the page
I have been thinking about installing a chat bot at home so I can talk to my Raspberry Pi server using a convenient interface: fire up a connection to a room and enter commands in plain old English. I want the bot to be able to recognise simple commands like "download a movie" or "set the alarm for 8:00am tomorrow". However I don't want to memorize a set of specific commands, or particular word orders - I want the interaction to feel like I am talking to a person, and not a machine.
Since my main hammer is Perl, I started looking around for natural language parser nails on CPAN, and came across something that looked promising: Lingua::LinkParser. It seems to be what I am looking for. It will parse a bit of text, and let me dive into the structure of the sentence to determine the meaning of it. So lets install it and give it a try.
My first step was to apt-cache search for liblingua and see if it is available. Strange, Debian usually has what I am looking for from CPAN as a package. I find when this happens it's probably that the software is unmaintained, so that should have been my first strong hint that this might become an Install saga!.
After some googling I found that the Perl module is an XS wrapper around a bit of software called "link-grammar" from the AbiWord project. So I visited the link grammar home page and looked for a download.
First I downloaded the zip archive of the master branch from the github repo, but the README launched straight into running the ./configure and my archive didn't have that file. I ran ./autogen.sh but encountered the following error:
./configure: line 17394: syntax error near unexpected token '2.0.0,'
./configure: line 17394: 'AX_PKG_SWIG(2.0.0, SwigF=yes, SwigF=no)'
Now type 'make' to compile link-grammar.
After more googling one of the developers commented somewhere to download an offical build instead of using the git repo, so I went back to the page and download the latest build tarball and extracted it. Sure enough this had a ./configure, so away we go:
./configure --disable-java-bindings --enable-perl-bindings --prefix=/opt/link-grammar
I wanted to install it in /opt/ so I could more cleanly blow it away if I didn't like it, and after seeing that it had not built Perl bindings when I first ran ./configure (which I assumed were needed for Lingua::LinkParser to work) I added the --enable-perl-bindings flag. After that, ./configure output ended with:
link-grammar-5.3.3 build configuration settings
prefix: /opt/link-grammar
C compiler: gcc -DUSE_SAT_SOLVER=1 -g -O2 -O3 -std=c11 -D_BSD_SOURCE -D_SVID_SOURCE -D_GNU_SOURCE -D_DEFAULT_SOURCE
C++ compiler: g++ -DUSE_SAT_SOLVER=1 -g -O2 -O3 -Wall -std=c++11
autopackage relocatable binary: no
Posix threads: no
Editline command-line history: no
UTF8 editline support: no
Java libraries: no
Java interfaces:
Swig interfaces generator: no
Perl interfaces: yes
Perl install location: /usr/local/lib/i386-linux-gnu/perl/5.20.2
Python interfaces: no
ASpell spell checker: no
HunSpell spell checker: no
HunSpell dictionary location:
Boolean SAT parser: yes
SQLite-backed dictionary: no
Viterbi algorithm parser: no
Corpus statistics database: no
RegEx tokenizer: no
Cool, so far so good. Not sure why it has no spell checker, but what the hell I can spell right!? Next step was to install local::lib so I could also blow away Lingua::LinkParser stuff if I wanted to. I boostrapped local::lib using these instructions. Now for the module install:
perl -MCPAN -Mlocal::lib -e 'CPAN::install(Lingua::LinkParser)'
But that expolded:
cc -c -I/usr/local/include/link-grammar/ -I/usr/local/include/ -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DVERSION=\"1.17\" -DXS_VERSION=\"1.17\" -fPIC "-I/usr/lib/i386-linux-gnu/perl/5.20/CORE" -DDICTIONARY_DIR=\".\" LinkParser.c
LinkParser.xs:5:27: fatal error: link-includes.h: No such file or directory
#include "link-includes.h"
It seemed the Perl module assumed the default install path for link-grammar. Fair enough, but still wanting to keep things separated I dug into the CPAN build directory and opened Makefile.PL to take a look. I changed this:
WriteMakefile(
'NAME' => "Lingua::LinkParser",
'VERSION_FROM' => "lib/Lingua/LinkParser.pm",
'DEFINE' => "-DDICTIONARY_DIR=\\\".\\\"",
## if your libs are in a nonstandard location, changes these, i.e.:
# 'LIBS' => "-L/dbrian/link-grammar-4.4.3/link-grammar/.libs/ -llink-grammar",
'LIBS' => "-L/usr/local/lib/ -llink-grammar",
'INC' => "-I/usr/local/include/link-grammar/ -I/usr/local/include/",
'OBJECT' => "",
);
To this:
WriteMakefile(
'NAME' => "Lingua::LinkParser",
'VERSION_FROM' => "lib/Lingua/LinkParser.pm",
'DEFINE' => "-DDICTIONARY_DIR=\\\".\\\"",
## if your libs are in a nonstandard location, changes these, i.e.:
# 'LIBS' => "-L/dbrian/link-grammar-4.4.3/link-grammar/.libs/ -llink-grammar",
'LIBS' => "-L/opt/link-grammar/lib/ -llink-grammar",
'INC' => "-I/opt/link-grammar/include/link-grammar/ -I/opt/link-grammar/include/ -I/usr/local/include/",
'OBJECT' => "",
);
Note: I needed to add both of those -I paths into /opt/ in order to get things to work. After running perl Makefile.PL, and then running make, I got the following error:
LinkParser.c: In function 'XS_Lingua__LinkParser_dictionary_delete':
LinkParser.c:257:9: error: void value not ignored as it ought to be
RETVAL = dictionary_delete(dict);
This had me stumped and googling for 1/2 an hour. Eventually I opened the source code in link-grammar to look at the definition of the dictionary_delete() function, and came to the conclusion that the Lingua::LinkParser XS code, and the version of link-grammar I had installed were not made for each other.
I looked up the Perl module on metacpan to see if I could see how old the Perl module was. In the change log I found the last entry was from 2014:
1.17 March 22 2014
- fixed build
- fixed dependency on dictionary_create
- fixed a broken test
- still need to remove dependencies on deprecated functions
Looks like my problem! The Perl module is from 2014, and my link-grammar is from 2015 - probably things have changed. So I go back to the abiword site and look at old versions. I grab one from the 2nd of Feburary 2014, hoping that the March 22nd Lingua::LinkParser was based off that. Nuked the newer version:
sudo rm -rf /opt/link-grammar/
Unpacked the new source, and ran the same ./configure:
configure: WARNING: unrecognized options: --enable-perl-bindings
link-grammar-4.8.6
prefix: /opt/link-grammar
C compiler: gcc -g -O2 -O3 -std=c99 -D_BSD_SOURCE -D_SVID_SOURCE -D_GNU_SOURCE
C++ compiler: g++ -g -O2 -O2 -Wall
autopackage: no
Posix threads: no
Editline command-line history: no
Java libraries: no
Java interfaces:
ASpell spell checker: no
HunSpell spell checker: no
HunSpell dictionary location:
Generate 'fat' linkages: no
Boolean SAT parser: no
Viterbi algorithm parser: no
Corpus statistics database: no
See that interesting message about the Perl bindings flag? Did the newer version have Perl bindings built already with the package? If only I could be bothered checking... After installing again in /opt/ I went back to the CPAN build directory, and did the dance:
perl Makefile.PL
make
make install
All looked good, it installed into my home directory and using the module produced no errors. Time to take it for a spin!
use strict;
use warnings;
use Lingua::LinkParser;
my $parser = Lingua::LinkParser->new;
my $sentence = $parser->create_sentence( "This is the turning point." );
my @linkages = $sentence->linkages;
for my $linkage (@linkages) {
print ($parser->get_diagram($linkage));
}
And there is is!!
link-grammar: Info: Dictionary found at /opt/link-grammar/share/link-grammar/en/4.0.dict
+-------------------Xp------------------+
| +--------Ost--------+ |
+------WV------+ +-------Ds------+ |
+---Wd---+-Ss*b+ | +---AN---+ |
| | | | | | |
LEFT-WALL this.p is.v the turning.n point.n .
+-------------------Xp------------------+
| +--------Osm--------+ |
+------WV------+ +-------Ds------+ |
+---Wd---+-Ss*b+ | +---AN---+ |
| | | | | | |
LEFT-WALL this.p is.v the turning.n point.n .
+-------------------Xp------------------+
| +--------Ost--------+ |
| | +-------Ds------+ |
+---Wd---+-Ss*b+ | +---AN---+ |
| | | | | | |
LEFT-WALL this.p is.v the turning.n point.n .
+-------------------Xp------------------+
| +--------Osm--------+ |
| | +-------Ds------+ |
+---Wd---+-Ss*b+ | +---AN---+ |
| | | | | | |
LEFT-WALL this.p is.v the turning.n point.n .
+-------------------Xp------------------+
| +--------Ost--------+ |
+------WV------+ +-------Ds------+ |
+---Wd---+-Ss*b+ | +---AN---+ |
| | | | | | |
LEFT-WALL this.p is.v the turning.g point.n .
+-------------------Xp------------------+
| +--------Osm--------+ |
+------WV------+ +-------Ds------+ |
+---Wd---+-Ss*b+ | +---AN---+ |
| | | | | | |
LEFT-WALL this.p is.v the turning.g point.n .
+-------------------Xp------------------+
| +--------Ost--------+ |
| | +-------Ds------+ |
+---Wd---+-Ss*b+ | +---AN---+ |
| | | | | | |
LEFT-WALL this.p is.v the turning.g point.n .
+-------------------Xp------------------+
| +--------Osm--------+ |
| | +-------Ds------+ |
+---Wd---+-Ss*b+ | +---AN---+ |
| | | | | | |
LEFT-WALL this.p is.v the turning.g point.n .
link-grammar: Info: Freeing dictionary en/4.0.dict
link-grammar: Info: Freeing dictionary en/4.0.affix
I have no idea what any of that means, but I feel I achived something. Now, why was I doing all that again??...