/ Projects

Description

Ewe and Fongbe are Niger–Congo languages, part of a cluster of related languages commonly called Gbe. Fongbe is the major Gbe language of Benin (with approximately 4.1 million speakers), while Ewe is spoken in Togo and southeastern Ghana by approximately 4.5 million people as a first language and by a million others as a second language. They are closely related tonal languages, and both contain diacritics that can make them difficult to study, understand, and translate.

Although those languages are at the core of the economic and social life of at least 3 major West African capital cities (namely Cotonou, Lome and Accra), they are today mostly spoken and very rarely written. Due to that fact (among other reasons), there is very little official or formal communication in those languages, leaving non-French/English speakers often unable to access critical facilities like education, banking, and healthcare. This challenge is part of an initiative that wishes to bring down the barriers between African local language speakers and modern society.

The objective of this challenge is to create a machine translation system capable of converting text from French into Fongbe or Ewe. You may train one model per language or create a single model for both. You may not use any external data, so a key component of this competition is finding a way to work with the available data efficiently.

This is a pioneer competition as far as low-resourced West African languages are concerned. A good solution would be a model that can be improved upon or used by researchers across the world to create APIs that can be integrated into day-to-day tools like ATMs, delivery applications etc., and help bridge the gap between rural West Africa and the modernized services.

This competition is one of five NLP challenges we will be hosting on Zindi as part of AI4D’s ongoing African language NLP project, and is a continuation of the African language dataset challenges we hosted earlier this year. You can read more about the work here.

About Takwimu Lab (takwimulab.gitlab.io)

TakwimuLab is an association of francophone west african who are professionals and enthusiasts about AI technologies. Our goal is to spread awareness about the challenges AI can help solve in our communities, disseminate knowledge and build solutions that can resolve real issues in our countries. Takwimu Lab is based in Benin.

Data

This is a parallel corpus dataset for machine translation from French to Ewe and French to Fongbe, languages from Togo and Benin respectively. It contains roughly 23 000 French to Ewe and 53 000 French to Fongbe parallel sentences, collected from blogs, tales, newspapers, daily conversations, webpages and annotated for neural machine translation. The collected sentences were preprocessed and aligned manually.

Variable definitions

ID : Unique identifier of the text
French : Text in French
Target_Laguauge: The target language
Target : Text in Fongbe or Ewe

Files available for download:

Train.csv – contains parallel sentences for training your model or models. There are 77,177 rows, of which 53,366 are French-Fongbe and 23,811 are French-Ewe
Test.csv- resembles Train.csv but without the Target column. This is the dataset on which you will apply your model(s).
SampleSubmission.csv – shows the submission format for this competition, with the ID column mirroring that of Test.csv and the ‘Target’ column containing your translation in Ewe or Fongbe. The order of the rows does not matter, but the names of the ‘ID’ must be correct.

Partners

AI4D-Africa; Artificial Intelligence for Development-Africa Network

Description

On social media, Arabic speakers tend to express themselves in their own local dialect. To do so, Tunisians use ‘Tunisian Arabizi’, where the Latin alphabet is supplemented with numbers. However, annotated datasets for Arabizi are limited; in fact, this challenge uses the only known Tunisian Arabizi dataset in existence.

Sentiment analysis relies on multiple word senses and cultural knowledge, and can be influenced by age, gender and socio-economic status.

For this task, we have collected and annotated sentences from different social media platforms. The objective of this challenge is to, given a sentence, classify whether the sentence is of positive, negative, or neutral sentiment. For messages conveying both a positive and negative sentiment, whichever is the stronger sentiment should be chosen. Predict if the text would be considered positive, negative, or neutral (for an average user). This is a binary task.

Such solutions could be used by banking, insurance companies, or social media influencers to better understand and interpret a product’s audience and their reactions.

About Icompass

iCompass is a Tunisian startup, created in July, 2019 and labelled startup act in August 2019. iCompass is specialized in the Artificial Intelligence field, and more particularly in the Natural Language Processing field. The particularity of iCompass is breaking the language barrier by developing systems that understand and interpret local dialects, especially African and Arab ones.

Partners

Description

Machine translation (MT) is a popular Natural Language Processing (NLP) task which involves the automatic translation of sentences from a source language to a target language. Machine translation models are very sensitive to the domain they were trained on which limit their generalization to multiple domains of interest like legal or medical domains. The problem is more severe in low-resource languages like Yorùbá where the most available datasets used for training are in the religious domain like JW300.

How can we train MT models to generalize to multiple domains or quickly adapt to new domains of interest? In this challenge, you are provided with 10,000 Yorùbá to English parallel sentences sourced from multiple domains like news articles, ted talks, movie transcripts, radio transcripts, software localization texts, and other short articles curated from the web. Your task is to train a multi-domain MT model that will perform very well for practical use cases.

The goal of this challenge is to build a machine translation model to translate sentences from Yorùbá language to English language in several domains like news articles, daily conversations, spoken dialog transcripts and books. Your solution will be judged by how well your translation prediction is semantically similar to the reference translation.

The translation models developed will assist human translators in their jobs, help English speakers to have better communication with native speakers of Yorùbá, and improve the automatic translation of Yorùbá web pages to English language.

About Masakhane

Masakhane is the open research, participatory, grassroots NLP initiative for Africans by Africans, with the aim of putting African research in NLP on the map, by holistically tackling the problems facing society. Founded in 2019, Masakhane has since garnered over 400 researchers from over 30 African countries, published state of the art research for over 38 African languages at various venues, and has built a thriving community. Masakhane’s participatory approach has enabled researchers without formal scientific training to contribute data, evaluations and models to published research, by focusing on lowering the barriers of entry.

Partner

Description

Twi is arguably the most recognizable Akan Language natively spoken in parts of southern and central Ghana, as well as parts of Cote d’Ivoire. By some estimates it has approximately 20 million native speakers [1]. It is a tonal language. It comprises at least four distinct dialects, namely Asante, Akuapem, Fante and Bono. Asante is arguably the most widely spoken and common dialect.

Building a database for Twi language in Africa

Pertinence

In practice, knowing this language alone allows one to navigate most parts of Ghana. You are likely to find someone who at the very least understands the language in every part of Ghana.

Example Sentences

English	Twi
What is going on here?	Ɛdeɛn na ɛrekɔ so wɔ ha?
Wake up	Sɔre
She comes here every Friday	Ɔba ha Fiada biara
Learn to be wise	Sua nyansa

Prior Work

The website and app Kasahorow [2] has a rather limited set of translations. The JW300 dataset [3] has some (over ½ million) extremely noisy English to (Akuapem) Twi parallel translation sentence pairs. A noisy Wikipedia is available [4], but the volume and quality leave much to be desired [4]. Some 700 sentence pairs are available in the TC Akan Corpus [9].

A recent study [5], which investigated the quality of these data sources in the context of FastText embeddings constructed on Twi, found them to be woefully insufficient. It is the only modern computing study of Twi that we are aware of. We have since replicated and slightly improved these FastText embeddings [6], trained and shared a variety of embeddings from the Transformers/BERT family through the HuggingFace model repo [7] and crowd sourced close to 1000 manually curated translation pairs. We have also developed a fairly decent English-Twi translator (transformer-based seq2seq model) which we are hoping to refine on the data that this collaboration yields. You can find more information on our official and github pages [8].

Researcher Profile: Paul Azunre

Paul Azunre holds a PhD in Computer Science from MIT and has served as a Principal Investigator on several DARPA research programs. He founded Algorine, a Research Lab dedicated to advancing AI/ML and identifying scenarios where they can have a significant social impact. Paul also co-founded NLP Ghana, an open source initiative focused on using NLP and Transfer Learning with Ghanaian and other low-resource languages. He frequently contributes to peer-reviewed journals and has served as a program committee member at some ICML workshops in AutoML and NLP. He is the author of the “Transfer Learning for NLP” book recently published by Manning Publications.

Researcher Profile: Lawrence Adu-Gyamfi

A subsea installation engineer by profession with a background in Aerospace engineering. Currently devoting the rest of my off-work time to contributing to the activities of NLP Ghana, assisting with the collection of data, preprocessing them and making them ready for use in the models we are testing internally. Serving as the NLP Ghana Director of Product, overseeing how the different teams of NLP Ghana work together.

Researcher Profile:Esther Appiah

Esther Appiah holds a BA in Modern Languages from the Kwame Nkrumah University of Science and Technology with a Diploma in French Studies from the Université D’Abomey Calavi, Centre Beninois des Langues Étrangères (CEBELAE) in Benin. She is currently pursuing an MPhil in Theoretical Linguistics at UiT, Norway. Her language specialties include French, English and Akan. She has a vast experience spanning various sectors/industries on language use and interface with core tasks on writing, proofreading, translation and researching. She works with the Ghana NLP as a data researcher and ultimately hopes to specialise in Computational Linguistics to help streamline NLP processes in underrepresented African languages in the digital space.

Researcher Profile: Felix Akwerh

Felix is currently enrolled in a Masters program in Computer Science at the Kwame Nkrumah University of Science and Technology. He augments his education with online classes and Machine Learning events. He is actively involved in the development of natural language processing with Ghana NLP. He co-authored a paper on Artificial Intelligence in Construction for submission. He holds a Bsc in Mathematics at the Kwame Nkrumah University. He worked with the UITS-KNUST where he helped build a transport system and other software projects. His research interest lies in Machine Learning and NLP, specifically in neural conversational models.

Researcher Profile: Salomey Osei

Salomey holds a Master of Philosophy in Applied Mathematics and an Msc in both Industrial Mathematics and Machine Intelligence. She is a recipient of Google and Facebook Scholarship, MasterCard Foundation Scholarship amongst others. She is the team lead for unsupervised methods for Ghana NLP and a co organizer for Women in Machine Learning and Data Science Accra chapter (WiMLDS). She is also passionate about mentoring students, especially females in STEM and her long term goal is to share her knowledge with others by lecturing.

Researcher Profile: Samuel Owusu

Samuel Owusu is currently working as a data scientist for the Ministry of Finance, Ghana. He holds a BSc in Information Technology from Ghana Technology University College. He was a team member of the group that won 1st prize of Ghana’s maiden national hackathon organised by the World Bank and Ministry of Water Resources and Sanitation. His Research interest lies in NLP – Automatic Speech Recognition for low resourced languages. He is involved in developing open source curriculums in Machine Learning and Computer Science for young girls. Samuel is a life-long learner.

Researcher Profile: Cynthia Amoaba

Cynthia Amoaba is a high school graduate from Chemu Senior High School and a student at the University For Development Studies. She’s an Ambassador and founder of the first Women In Stem (WiSTEM) chapter in Ghana.She also founded the STEM club in her high school and looks forward to extending it to schools in deprived areas. Currently, she tutors high school students in her community in Physics and Maths and helps train school dropouts in beads and soap making. She’s a science enthusiast and looks forward to learning more through her involvement in the development of NLP with Ghana-NLP.

Researcher Profile: Salomey Afua Add

Salomey Afua Addo is the founder of Lighted Hope, a Non Governmental Organization that seeks to promote literacy and coding skills among children living in slums in Ghana. She holds an MSc in Mathematical Sciences from the African Institute for Mathematical Sciences and a certificate in business management from the European School of Management and Technology, Berlin. She is the coding instructor for The Love Academy in the USA. Currently, she serves as a volunteer at Ghana NLP, and she plays a vital role in collecting and preprocessing data for the data team at Ghana NLP. Salomey Afua Addo lives a purpose driven life.

Researcher Profile: Edwin Buabeng-Munkoh

Edwin Buabeng-Munkoh is currently working as a Software Engineer at Huawei Technologies Ghana Limited. He holds a BSC in Computer Engineering from Kwame Nkrumah University of Science and Technology. He is enrolled in the Data Science Mentorship program with Notitia AI. He is actively involved in the development of natural language processing with GhanaNLP. He serves as a volunteer at Ghana NLP where he helps with preprocessing data for the data team. Along with his daily work he has enrolled and completed multiple online courses on Data Science, AI and NLP. His research interest lies in Machine Learning, NLP and Computer Vision. He plans to help build a world where language is not a barrier in education and good healthcare

Researcher Profile:Nana Boateng

Nana Boateng holds a PhD. in Statistics from The University of Memphis. He has three masters degrees in Statistics, Mathematics and Economics. He has worked as a Data Scientist for Companies such as Fiat Chrysler Automobiles, Nice Systems Inc and Baptist Memorial Hospital. He is interested in application of mathematics, statistics and economics principles in solving problems in healthcare, finance and several other industries. He has several peer-reviewed publications to his name. He is the founder of Rest Analytics which advises companies on how to apply machine learning to increase efficiency and productivity. He contributes to GhanaNLP in the area of supervised learning.

Partners

References

https://en.wikipedia.org/wiki/Twi
https://www.kasahorow.org/
Z. Agic et. al., JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages, ACL Proceedings
https://ak.wikipedia.org/
J. Alibi et al., Massive vs. Curated Word Embeddings for Low-Resourced Languages. The Case of Yorùbá and Twi, LREC Proceedings 2020
https://medium.com/swlh/ghana-nlp-computational-mapping-of-ghanaian-languages-edf60c56bcce
https://huggingface.co/Ghana-NLP
https://ghananlp.github.io/
https://www.researchgate.net/publication/323998547_TypeCraft_Akan_Corpus_Release_10

Disclaimer

The designations employed and the presentation of material on these map do not imply the expression of any opinion whatsoever on the part of the Secretariat of the United Nations concerning the legal status of any country, territory, city or any area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Final boundary between the Republic of Sudan and the Republic of South Sudan has not yet been determined. Final status of the Abyei area is not yet determined.

Description

The Ganda language or Luganda is a Bantu language spoken in the African Great Lakes region. It is one of the major languages in Uganda, spoken by more than eight million Baganda and other people principally in central Uganda, including the capital Kampala of Uganda. It belongs to the Bantu branch of the Niger-Congo language family. Typologically, it is a highly-agglutinating, tonal language with subject-verb-object, word order, and nominative-accusative morphosyntactic alignment [1].

Language profile: Luganda

Building a database for Luganda language in Africa

With about six million first-language speakers in the Buganda region and a million others fluent elsewhere, it is the most widely spoken Ugandan language. As a second language, it follows English and precedes Swahili [1].

The language is used in all domains: education, media, telecommunication, trade, entertainment and in religious centres [2]. It has been used at lower institutions as pupils begin to learn English and until the 1960s, Luganda was also the official language of instruction in primary schools in Eastern Uganda [1]. It is also among the languages that have been tabled in the East African parliament to be selected as the official language for the East Africa Community [2].

Existing Work

As the use of the Luganda language is drastically growing across the different sectors from formal to informal, there has been work done on building a Luganda corpus and developing NLP models such as a Luganda text to speech machine [3]; English noun phrase to Luganda translator [4], smart Luganda language translator – given a source text in English it translates it to Luganda automatically [5]. To broaden access to search, a Luganda interface was launched for Google web search [6]. However, some of these applications have been developed based on minimal data.

In terms of language resources, there exists the Luganda Bible [7] which is an online Bible from the Word Project [13] and other religious books from the Jehovah’s Witnesses [12]. There exist some good online Luganda dictionaries like the globe dictionary [10], learn Luganda [9], Luganda phrasebook [9], learn Luganda concise [11] dictionary and Luganda Dictionary [8]. However, most of the available dictionaries are copyrighted and contain just a few word extracts in the language which results in a small representation for a language like Luganda where new words are being created and spoken every year.

Quite recently, a drive has been made by a team of researchers from the Makerere AI Lab [15] to add Luganda to the Common voice platform [14] and it is anticipated that through this project, a large voice dataset for building voice recognition models for Luganda will be generated.

Example of Sentence in Luganda

Luganda: Aboomukyalo bafuna nnyo mu by’obulimi.

English: The people in rural areas benefit a lot from agriculture.

Conclusion

With the regional integration of the East African Community in place, the use of the Luganda language has stretched boundaries from Uganda to the East African community, because most of the native speakers of this language are actively participating in this cooperation. It has been used to support inter-ethnic communication [2]. This, however, stretches beyond East Africa on the other hand.

Therefore, there is a need to build a robust Luganda dataset, which can be made publicly available so that different researchers can use it to build downstream applications such as machine translators, speech recognition machines, chatbots, virtual assistants, sentiment analytic models, in ensuring that information is accessible to all and also addressing some of the local-contextual problems with in the society.

Researcher Profile: Joyce Nakatumba-Nabende

Joyce Nakatumba-Nabende is a lecturer in the Department of Computer Science in Makerere University. She is also the head of the Makerere Artificial Intelligence and Data Science Lab in the College of Computing and Information Sciences. She obtained a PhD in Computer Science from Eindhoven University of Technology, The Netherlands. Her current research interests include Natural Languages Processing, Machine Learning, and Process Mining and Business Process Management. She is co-author of more than 20+ papers published in peer-reviewed international journals and conferences. She has supervised several PhD and Masters students in the field of Computer Science and Information Systems. She is a member of several international AI bodies that include Open for Good Alliance, Feministic AI Network and UN Expert Group Recommendation 3C Group on Artificial Intelligence.

Researcher Profile: Andrew Katumba

Andrew Katumba is a Lecturer in the Department of Electrical and Computer Engineering as well as a senior researcher with netLabs!UG, a research Center of Excellence in Telecommunications and Networking both in the College of Engineering, Design, Art & Technology (CEDAT), Makerere University. Andrew champions the research and applied Artificial Intelligence (AI) activities at netLabs!UG as the lead for the Marconi Society Machine Learning Lab. Andrew holds a PhD in Photonics and Machine Learning from the Gent University, Belgium. He has co-authored 50+ publications in peer-reviewed international journals and conferences and holds 2 patents in neuromorphic computing.

Researcher Profile: Jonathan Mukiibi

Jonathan Mukiibi is a computer science practitioner with a background in software engineering,linguistics, machine learning, big data and natural language processing. Over the past years he has been involved in artificial intelligence based projects like satellite image analysis, radio mining, social media mining, ambulance tracking and traffic which have been successfully implemented to solve real world problems in developing communities. Currently, he is pursuing a Masters in Computer Science at Makerere University where he is also doing research work at the AI and Data Science Research Lab. He is actively working on different NLP tasks but majorly doing research in end-to-end topic classification models for crop pests and disease surveillance from radio recordings.

Researcher Profile: Claire Babirye

Claire Babirye is a computer science professional with vast experience in different computing modules: from computer networks, computer security, network monitoring to machine learning, data science, natural language processing, deep learning technologies and use of technology for improved service delivery. She is a Research Assistant at the AI and Data Science Research Lab Makerere University and her role is to: tap into the revolution to obtain more and better data so as to support development work and humanitarian; support data analytics and visualization to generate patterns on insights on the data; develop machine learning models for classification. Within the domain of NLP, she has worked on tasks that involve: sentiment analysis on social media data and text classification to identify topics of interest from the farmer agricultural data.

References

https://en.wikipedia.org/wiki/Luganda
https://www.open.edu/openlearn/languages/more-languages/linguistics/english-squeezing-out-local-languages-uganda
Nandutu, I., & Mwebaze, E. (2020). Luganda Text-to-Speech Machine. arXiv preprint arXiv:2005.05447.
https://www.researchgate.net/publication/338036914_Model_for_Translation_of_English_Language_Noun_Phrases_to_Luganda
https://www.researchgate.net/publication/323682143_Smart_Luganda_Language_Translator
https://africa.googleblog.com/2009/07/how-volunteer-translators-impact-local.html
https://play.google.com/store/apps/details?id=com.LugandaBible&hl=en
https://web.archive.org/web/20080120211744/http://www.cbold.ddl.ish-lyon.cnrs.fr/CBOLD_Lexicons/Ganda.Snoxall1967/Text/Ganda.Snoxall1967.txt
https://learn-luganda.com/
https://glosbe.com/lg/en/
https://learnluganda.com/concise
https://wol.jw.org/lg/wol/h/r138/lp-lu
https://www.wordproject.org/bibles/lug/index.htm
https://commonvoice.mozilla.org/lg
http://www.air.ug/natural-language-processing

Partners

Disclaimer

Motivation

Keyword spotting refers to the task of learning to detect spoken keywords. It interfaces all modern voice-based virtual assistants on the market: Amazon’s Alexa, Apple’s Siri, and the Google Home device. Contrarily to speech recognition models, keyword spotting doesn’t run on the cloud, but directly on the device. This sets up a natural constraint on the model size, energy consumption, and compute efficiency of the model because often the hardware devices have limited memory and limited computing power.

The prerequisite to perform such a task with African languages would be the creation of a dedicated dataset. Indeed, African languages account for 30.15% of the 7111 living languages (Orife et al. 2020), which provide great diversity. Unfortunately, they are barely represented in natural language processing (NLP) research.

The motivation of this proposal is to extend the Speech commands dataset (Warden 2018) with African languages. In particular, we are going to focus on 6 Senegalese languages: Wolof, Poular, Sérère, Mandingue, Diola, Soninké. Those are the most spoken languages in Senegal, and the first to be codified (given a written form) in 1971. There are two distinctions between a language and a dialect. In the case of Wolof, Poular, Mandingue, and Soninké there are many spoken dialects, each belonging to particular regions in Senegal.

Moreover, in this case, two speakers of different dialects within the same language may understand each other. In other words, the language is of common understanding for all the speakers of regional dialects. In the case of Sérère and Diola however, regional dialects differ dramatically, to the point that there is no common understanding for speakers of different dialects of the same language. In this case, the chosen language is the most spoken dialect within the language, the Sérère-Sine, and the Diola-Fogny.

Goals and outcomes

The Speech commands dataset (Warden 2018) included only a limited vocabulary composed of around twenty common words at its core. These included the digits from zero to nine, and seventeen words that would be useful as commands in IoT or robotics applications; “Yes”, “No”, “Up”, “Down”, “Left”, “Right”, “On”, “Off”, “Stop”, and “Go”, “Backward”, “Forward”, ”Bed”, “Bird”, “Cat”, “Dog”, “Tree”. This dataset has been collected specifically for keyword spotting referring to the task of learning to detect keywords. In short, the objective of this proposal is three-fold:

To produce open-source software dedicated to collecting data in an African context.
To build up an open-source extension of the Speech Commands dataset with Senegalese languages.
To document the dataset creation process and publish it, so that it could be reproduced in other countries. The interest here is to make the process reproducible in the African context

Concept and methodology

The first goal of the project is to produce open-source software dedicated to collect speech data in the African context. To do that, we are going to extend Common Voice so that it could mirror Amazon Mechanical Turk in an African context. Indeed, Common Voice is open-source software started by Mozilla to create a free database for speech recognition software. Common Voice natively allows users to create an account and to upload speech for a given text.

The account allows us to be sure that every recording comes from a different person. In addition, it also allows for a peer to peer validation, which could help for scaling. Amazon Mechanical Turk allows researchers to collect data from the internet by paying users a small amount of money, but is closed and only works with traditional payment solutions such as credit card or Paypal. However, mobile money is pervasive in most African countries. Thus, the main challenge for this deliverable is to connect Common Voice to mobile money solutions in Senegal, but the end goal would be to allow contributions in an open-source fashion to integrate with other mobile money actors in Africa like M-PESA.

For that purpose we made a successful partnership with Baamtu. As a technical partner, they are going to take care of all the cost relative to the web application development and deployment. Thierno Diop, one of the members of GalsenAI and Lead Data scientist at Baamtu, has won the AI4D-African Language Dataset Challenge in November 2019 with this methodology.

The second deliverable is to collect a dataset composed of 1000 recordings of the vocabulary in each language. For that purpose, we have made a strong multidisciplinary partnership with a team of linguists that is going to help to translate each word of the vocabulary in the six languages:

Wolof: Dr. Mamour Dramé Teacher Assistant, Department of Linguistics and Language Science, Faculty of Humanities, Cheikh Anta Diop University.
Poular: Mr. Moctar Baldé, Ph.D. Student in Linguistic and Teacher Assistant, Department of Linguistic and Language Science, Faculty of Humanities, Cheikh Anta Diop University.
Sérère : Mr. Edouard Diouf, Ph.D. Student in Linguistic and Teacher Assistant, Department of Linguistic and Language Science, Faculty of Humanities, Cheikh Anta Diop University.
Diola-Fogny : Mr. Pascal Assine, Ph.D. Student in Linguistic and Teacher Assistant, Department of Linguistic and Language Science, Faculty of Humanities, Cheikh Anta Diop University.
Mandingue: Dr. Mamadou Dabo, Teacher Assistant, Department of Linguistics and Language Science, Faculty of Humanities, Cheikh Anta Diop University.
Soninké: Mr. Almamy Konaté, Consultant Translator in Soninké.The dataset is going to be accessible on the GitHub page of the open-source project.

Finally, the third deliverable is to document the whole process and publish it in a technical report.

Greater narrative and ambition

The full impact of this project can be decomposed into two dimensions. (1) An open-source software with a reproducible process to create a dataset specifically in the African context. (2) A modeling challenge: keyword spotting with African languages that focuses on a practical machine learning task in a low resource setting. We argue that keyword spotting with African languages could have a widespread adoption within the African machine learning (ML) community and is, therefore, more likely to lead to breakthroughs and original contributions from Africa.

Low resource: Keyword spotting is a natural task to experiment with smaller models, requiring fewer resources to train (Warden 2018). It is relevant for the African ML research community who might work in limited-resource settings because of the systemic lack of funding in African research laboratories, besides, the majority of undergrad students might be under-equipped to run large models. Thus, alleviating the constraint of requiring a lot of computation and big datasets could harness the creativity of the African ML community. Indeed this trend has been acknowledged with the Practical ML for Developing Countries Workshop at ICLR 2020. We think that the current project reinforces this trend and could lead the African ML community toward academic leadership in resource constraint ML. Indeed, datasets and benchmarks have often played a significant role in pushing efforts toward discoveries, for instance, we can think of MNIST and ImageNet for that purpose.
Relevant and practical: We think that working with African language could have a wider adoption by the African ML community because we think that people have a genuine interest in developing tools for their language due to the intrinsic social value of languages. Moreover, as Africa has a strong oral tradition (Iwuji, 1989), we think that the general public could greatly benefit from services developed with their native language, which would encourage ML developers towards more innovation, which could lead to a virtuous circle of value creation.
Reproducibility and scalability: (1) open-source software allows any developer to adapt the software to feature local mobile money solutions. (2) Our methodology to select languages can then be used as a reference in that country to collect data for new languages. (3) The Github page of the project would then display a link to every new dataset created with the software. (4) Finally, as the validation procedure is expected to be performed in a peer to peer fashion. The data collection process is expected to scale with respect to the number of speakers willing to be recorded, and the number of people willing to validate the words from a given language.

Personnel

Jean Michel Ahmath Sarr – PhD Student in Computer Science, Department of Mathematics and Computer Science, Faculty of Science, Cheikh Anta Diop University. GalsenAI – IndabaX Senegal organizer
Daouda Tandiang Djiba – Data Scientist – BICIS Group BNP Paribas – GalsenAI – IndabaX Senegal organizer
Thierno Diop – Lead Data Scientist at Baamtu, cofounder of GalsenAI and Zindi Ambassador in Senegal – IndabaX Senegal organizer
Derguene Mbaye – Engineering Student in Telecoms & networks at Ecole Superieure
Polytechnique de Dakar, GalsenAI – IndabaX Senegal organizer
Elias waly Ba – Lead Data Scientist at Air Senegal – GalsenAI – IndabaX Senegal organizer
Ousseynou Mbaye – PhD Student in Computer Science, Department of Applied Sciences, Communication and Technologies, Alioune Diop University of Bambey – GalsenAI – IndabaX Senegal Organizer
Dr Mamour Dramé, Teacher

Description

This dataset is part of a 3-4 month Fellowship Program within the AI4D – African Language Program, which was conceptualized as part of a roadmap to work towards better integration of African languages on digital platforms, in aid of lowering the barrier of entry for African participation in the digital economy.

This particular dataset is being developed through a process covering a variety of languages and NLP tasks, in particular Machine Translation of Ewe.

Language profile: Ewe

Language profile for Ewe

Overview

Ewe (Èʋe or Èʋegbe [èβeɡ͡be]) is a Niger–Congo language spoken in Togo and southeastern Ghana by approximately 4.5 million people as a first language and a million or so more as a second language.[1] Ewe is part of a cluster of related languages commonly called Gbe; the other major Gbe language is Fon of Benin. Like many African languages, Ewe is tonal.

Pertinence

In Togo and Ghana where ewe is heavily spoken, it is the main communication medium in major economic hubs especially in Togo where it is the most spoken language in the capital city Lome and also one of the two national languages of the country. In Ghana, ewe is part of the 11 government-sponsored languages apart from the official language english.[2] In 2020, the majority of children growing up in major cities in the Togo still picks up as their first language, a dialect of ewe depending on the region of the country they are from. The majority of the speakers at this day can speak ewe but not write it with the appropriate alphabet and orthography. The written communications in ewe usually happen using the english/french alphabet to write the sounds made by the words. Some schools in the capital offer ewe courses at secondary school level but those are generally optional and focus only on basics.

Nevertheless, in Togo, while the communication in schools and formally registered companies takes place in french, ewe remains the most used language in critical settings such as :

Market places
Medical centers
In apprenticeship for a major array of occupations such as hairdressing, tailoring, engine repairing, carpenting, agriculture among other manual jobs that make up 90% of the jobs and 30% of the GDP of the country. [3]
at police stations
in banking or telecommunications agencies
in shops and restaurants

Existing Work

Apart from sparse efforts of actors in the academic and literary fields and from some associations, there has not been any federated effort from the togolese government. Wycliffe-togo [4] is however one of the most prominent associations in the country organizing events and doing work to promote local languages. There exist a few ewe-english/french dictionaries online but the most popular ones remain the glosbe dictionary on the web [5], the Kasahorow Evegbe English Dictionary [6] and the mobile Ewe Dictionary [7] on the Android play store. In the academic world, a lot of work has been done especially regarding the tone, the syntax but also on other aspects such as the anthropological, lexicographical and phonological domains by both foreign and local (ghanain) researchers. [1]

Example of sentence in Ewe

Ewe : Ne ati aɖe le nya dim ɣesiaɣi le fíá wo ŋuti la, mumu ye le dzrom.

English : A tree which provokes axes wishes to be cut down.

Researcher Profile: Kevin Degila

Kevin is a Machine Learning Research Engineer at Konta, an AI startup based in Casablanca. he holds an engineering degree in Big Data and AI and it’s currently enrolled in a PhD program focused on business document understanding at Chouaib Doukkali University. In his day to day activities, Kevin train, deploy and monitor in production machine learning models. With his friends, they lead TakwimuLab, an organisation working on training the next young, french speaking, west africans talents in AI and solving real-life problems with their AI skills. In his spare time, Kevin also create programming and AI educational content on Youtube and play video games.

Researcher Profile: Momboladji Balogoun

Momboladji BALOGOUN is the Data Analyst of Gozem, a company providing ride-hailing and other services in West and Central Africa. He is a former Data Scientist at Rintio, an IT startup based in Benin, that uses data and AI to create business solutions for other enterprises. Momboladji holds a M.Sc. degree in Applied Statistics from ICMPA UNESCO Chair, Cotonou, and migrated to the Data Science field after having attended a regional Big Data Bootcamp in his country Benin. He aims to pursue a Ph.D. program on low resources languages speech to speech translation. Bola created Takwimu LAB in August 2019, and he leads it currently with 3 other friends in order to promote Data Science in their countries, but also the creation and the use of AI to solve real-life problems in their communities. His hobbies are: Reading, Documentaries, and Tourism.

Researcher Profile: Godson Kalipe

Godson started in the IT field with software engineering with a specialization on mobile applications. After his bachelor in 2015, he worked for a year as web and mobile application developer before joining a master in India in Big Data Analytics. His master thesis consisted comparative analysis of international news impact on economic indicators of African countries using news Data, Google Cloud storage and visualization assets. After his Master,

in 2019, he gained a first experience as Data Engineer creating data ingestion pipelines for real time sensor data at Activa Inc, India. He parallely has been working with Takwimu Lab on various projects with the aim of bringing AI powered solutions to common african problems and make the field more popular in the west African francophone industry.

Researcher Profile: Jamiil Toure

Jamiil is a design engineer in electrical engineering from Ecole Polytechnique d’Abomey-Calavi (EPAC), Benin in 2015 and a master graduate in mathematical sciences from African School of Mathematical Sciences (AIMS) Senegal in 2018. Passionate of languages and Natural Language Processing (NLP), he contributes to the Masakhane project by working on the creation of a dataset for the language Dendi.

Meanwhile, he complements his education on NLP concepts via online courses, events, conferences for a future research career in NLP. With his friends at Takwimu Lab they work at creating active learning and working environments to foster the applications and usages of AI to tackle real-life problems. Currently, Jamiil is a consultant in Big Data at Cepei – a think tank based in Bogota that promotes dialogue, debate, knowledge and multi-stakeholder participation in global agendas and sustainable development.

Partners

Disclaimer

Description

This particular dataset is being developed through a process covering a variety of languages and NLP tasks, in particular Machine Translation of Fongbe.

Language profile: Fongbe

Language profile for Fongbe

Overview

Fon or fɔ̀ngbè is a low resource language, part of the Eastern Gbe language cluster and

belongs to the Volta–Niger branch of the Niger–Congo languages. Fongbe is spoken in Nigeria, Togo and mainly in Benin by approximately 4.1 million speakers. Like the other Gbe languages, Fongbe is an analytic language with an SVO basic word order. It’s also a tonal language and contains diacritics which makes it difficult to study. [1]

The standardized Fongbe language is part of the Fongbe cluster of languages inside the Eastern Gbe languages. In that cluster, there are other languages like Goun, Maxi, Weme, Kpase which share a lot of vocabulary with the Fongbe language. Standard Fongbe is the primary target of language planning efforts in Benin, although separate efforts exist for Goun, Gen, and other languages of the country. To date, there are about 53 different dialects of the Fon language spoken throughout Benin.

Pertinence

Fongbe holds a special place in the socio economic scene in Benin. It’s the most used language in markets, health care centers, social gatherings, churches, banks, etc.. Most of the ads and some programs on National Television are in Fongbe. French used to be the only language of education in Benin, but in the second decade of the twenty first century, the government is experimenting with teaching some subjects in Benin schools in the country’s local languages, among them Fongbe.

Example of Fongbe Text:

Fongbe : Mǐ kplɔ́n bo xlɛ́ ɖɔ mǐ yí wǎn nú mɛ ɖevo lɛ

English : We have learned to show love to others [3]

Existing Work

Some previous work has been done on the language. There are doctorate thesis, books, French to Fongbe and Fongbe to French dictionaries, blogs and others. Those are sources for written fongbe language.

Researcher Profile: Kevin Degila

Researcher Profile: Momboladji Balogoun

Researcher Profile: Godson Kalipe

Researcher Profile: Jamiil Toure

Partners

Disclaimer

Description

This particular dataset is being developed through a process covering a variety of languages and NLP tasks, in particular Machine Translation of Yoruba.

Language profile: Yoruba

Language profile for Yoruba

Overview

The Yorùbá language is the third most spoken language in Africa, and is native to the south-western Nigeria and the Republic of Benin in West Africa (as shown in Figure 1). It is one of the national languages in Nigeria, Benin and Togo, and it is also spoken in other countries like Ghana, Côte d’Ivoire, Sierra Leone, Cuba, Brazil and by a significant Yorùbá diaspora population in the US and United Kingdom mostly from the Nigerian ancestry. The language belongs to the Niger-Congo family, and is spoken by over 40 million native speakers [1].

Yorùbá has several dialects but the written language has been standardized by the 1974 Joint Consultative Committee on Education [2], it has 25 letters without the Latin characters (c, q, v, x and z) and with additional characters (ẹ , gb, ṣ , and ọ). There are 18 consonants (b, d, f, g, gb, j, k, l, m, n, p, r, s, s., t, w y), and 7 oral vowels (a, e, ẹ , i, o, ọ , u). Yorùbá is a tonal language with three tones: low, middle and high.

These tones are represented by the grave (“\”), optional macron (“- ”) and acute (“/”) accents respectively. These tones are applied on vowels and syllabic nasals, but the mid tone is usually ignored in writings. The tones are represented in written texts along with a modified Latin alphabet. A few alphabets have underdots (i.e. “ẹ ”, “ọ ”, and “ṣ”), we refer to the tonal marks and underdots as diacritics. It is important to note that tone information is needed for correct pronunciation and to have the meaning of a word [2, 3].

As noted in [4], most of the Yorùbá texts found in websites or public domain repositories either use the correct Yorùbá orthography or replace diacriticized characters with un-diacriticized ones.

Oftentimes, articles written online including news articles1 like BBC and VON ignore diacritics. Ignoring diacritics makes it difficult to identify or pronounce words except they are in a context. For example, owó (money), ọwọ̀ (broom), òwò (business), ọ̀wọ̀ (honour), ọwọ́ (hand), and ọ̀wọ́ (group) will be mapped to owo without diacritics.

Existing work

Due to the problem with the diacritics in Yorùbá language, it has greatly reduced the amount of available parallel texts that can be used for many NLP tasks like machine translation. This has led to research on automatically applying diacritics to Yorùbá texts [5, 6], but the problem has not been completely solved. We will divide the existing work on Yorùbá language into four categories:

Automatic Diacritics Application

The main idea for the automatic diacritic application (ADA) model is to predict the correct diacritics of a word based on the context it appears. We can make use of a sequence-to-sequence deep learning model like Long Short Term Memory networks (LSTM) [7] to achieve this task.

The task is similar to a machine translation task where we need to translate from a source language to a target language, ADA takes a source text that is non-diacriticized (e.g “bi o tile je pe egbeegberun ti pada sile”) and outputs target texts with diacritics (e.g. “bí ó tilẹ̀ jẹ́ pé ẹgbẹẹgbẹ̀rún ti padà síléé”). The first attempt of applying deep learning models to Yorùbá ADA was by Iroro Orife [5].

They proposed a soft-attention seq2seq model to automatically apply diacritics to Yorùbá texts, their model was trained on the Yorùbá bible, Lagos-NWU speech corpus and some language blogs. However, the model does not generalize to other domains like dialog conversation and news domain because the majority of the texts are from the Bible. Orife et al [6] recently addressed the issue of domain-mismatch by gathering texts from various sources like conversation interviews, short stories and proverbs, books, and JW300 Yorùbá texts but they evaluated the performance of the model on the news domain (i.e Global Voices articles) to measure domain generalization.

Word Embeddings

Word embeddings are the primary features used for many downstream NLP tasks. Facebook released FastText [8] word embeddings for over 294 languages 2 but the quality of the embeddings are not very good. Recently, Alabi et. al [9] showed that Facebook’s FastText embeddings for Yorùbá gives a lower performance in word similarity tasks, which indicates that they would not work well for many downstream NLP tasks. They released a better quality FastText embeddings and contextualized BERT [10] embeddings obtained by fine-tuning multi-lingual BERT embeddings.

Datasets for Supervised Learning Tasks

Yorùbá, like many other low-resourced languages, does not have many supervised learning datasets such as named entity recognition (NER), text classification and parallel sentences for machine translation. Alabi et al. [9] created a small NER dataset with 26K tokens. Through the support of AI4D 3 and Zindi Africa 4, we have created parallel English-Yorùbá dataset for machine translation and news title classification dataset for Yorùbá from articles crawled from BBC Yorùbá 5. The summary of the AI4D dataset creation competition is in [11].

Machine Translation

Commercial machine translation models like Google Translate 6 exist for Yorùbá to other languages but the quality is not very good because of the diacritics problem and the small amount of data available to train a good neural machine translation (NMT) model. JW300[12] based on Jehovah Witness publications is another popular dataset for training NMT models for low-resource African languages, it has over 10 million tokens of Yorùbá texts. However, the NMT models trained on JW300, do not generalize to other non-religious domains. There is a need to create more multi-domain parallel datasets for Yorùbá language.

Researcher Profile: David Adelani

David Ifeoluwa Adelani is a doctoral student in computer science at Spoken Language Systems Group, Saarland Informatics Campus, Saarland University, Saarbrücken, Germany. His current research focuses on the security and privacy of users’ information in dialog systems and online social interactions.

He is also actively involved in the development of natural language processing datasets and tools for low-resource languages, with special focus on African languages. He has published a few papers in top Web technology, language and speech conferences including The Web Conference, LREC, and Interspeech.

During his graduate studies, he conducted research on social computing at the Max Planck Institute of Software Systems, Germany and on fake review detection at the National Institute of Informatics, Tokyo, Japan. He holds an MSc in Computer Science from the African University and Science and Technology, Abuja, Nigeria and a BSc in Computer Science from the University of Agriculture, Abeokuta, Nigeria.

Partners

Disclaimer

Description

This particular dataset is being developed through a process covering a variety of languages and NLP tasks, in particular Document Classification datasets of Chichewa.

Language profil: Chichewa

Language profile for Chichewa

What is Chichewa?

Chichewa is part of the Niger-Congo Bantu group and it is one of the most spoken indigenous languages of Africa. Chichewa is both an individual dialect and a language group as we shall discuss in this short article.

The language, Chichewa, also written as Cichewa, or, in Zambia, Cewa, is the native language of the Chewa. The word ‘chi’ or ‘ci’ is a Bantu prefix used for the tribal name, designating the language rather than the geographical region of the tribe. The word Chewa is the name of a group of people. Chichewa is called Chinyanja, for example in Zambia and Mozambique. Chinyanja was also the old name for the language in Malawi, before the country became a Republic. During that time, as a British Protectorate, Malawi was called Nyasaland.

Chichewa, with the code ‘ny’ is also one of the 13 African languages with a Google automatic translation. The code ‘ny’ was most likely chosen because the language was known first as Chinyanja. This probably reflects the availability of written text in Chichewa compared to other African languages. However, as we will discuss in this article, there are several dialects of Chichewa which differ from each other in noticeable ways. I do not know whether this was taken into account for the text used in the machine language models by Google. But this is a whole new interesting topic in itself!

Who are the Chewa?

The Chewa are a Bantu speaking people, traditionally described as the descendants of the Maravi, who in the 16th (some say, in the 14th) century migrated to the present day Malawi from the region now called Congo-Kinshasa. Most of what we know about the migrations of the Cewa come from oral tradition. Samuel Nthara collected some of the oral traditions in his book Mbiri ya Achewa, published in 1944. The name Maravi first appeared in Portuguese documents in 1661.

Nowadays, some of the well known districts in Malawi where the Chewa live are: Mchinji, Lilongwe, Kasungu, Nkhotakota, Dowa and Dedza. The consensus is that the Chewa of the mainland kept their name as Chewa and lived mainly in the Central Region. The Manganja are the Chewa who settled in the Southern region. And some Chewa groups who settled at the lake or around the Shire River in the south are called Nyanja. Man’ganja (or Maganja) is southern Chichewa as opposed to the language spoken in the Central Region (which was also called Western Chichewa / Nyanja). There are phonetical, grammatical and vocabulary differences between these dialects.

Where is Chichewa spoken?

In Malawi, Chichewa is widely understood. It was declared the national language in 1968 and it is viewed as a symbol of national unity by diverse groups. In Mozambique it is spoken especially in the provinces of Tete and Niassa, where it is referred to as Chinyanja. In Zambia, it is spoken in Lusaka and in the Eastern Province (the language is referred to as Nyanja). The language spoken in Lusaka is sometimes called town-Nyanja as opposed to the Nyanja spoken in rural areas in other parts of Zambia, where it is referred to as deep-Nyanja. Nyanja is the language of the Police and the Army. In Zimbabwe, according to some estimates, Chichewa is the third most widely used language after Shona and Ndebele. There is a sizable community of descendents from those who migrated to this area from Nyasaland during colonial times to work in the mines.

Chichewa is spoken in South Africa. There are a significant number of migrants from Malawi who work in mining, as domestic workers or in other industries. There are radio services in Chichewa in Malawi, Zambia, South Africa and even in Ethiopia.

How many people speak the language?

According to sources quoted in Wikipedia, there are 12 million native speakers of Chichewa. A similar number is mentioned on the Joshua project website and includes Chichewa speakers from 8 countries of the world. This number seems then to refer to all the people who identify themselves as Chewa, Nyanja and Manganja, as these, according to the Malawi Population Census of 2018, make about 40% of the population in Malawi. However, in Malawi, the large ethnic groups of Lomwe, Yao and Ngoni have over the course of time adopted Chichewa as their native language.

It is the case that the number of people understanding and using Chichewa is much higher than the 12 million native speakers. Like Swahili, Chichewa is considered by some a universal language, a common skill enabling people of varying tribes and those living in Malawi, Zambia, Mozambique to communicate without following the strict grammar of specific local languages. In Zambia, many of those whose mother tongue is now Chinyanja have come to consider themselves Ngoni; Nyanja is a lingua franca, being spoken by the police and the administration.

The Need for Datasets in Chichewa

As discussed, seven important facts provide impetus to the initiative to develop data set for Chichewa: (1) Chichewa is an important African language, (2) it is representative of the Niger Congo Bantu group of languages, (3) it is widely spoken, (4) it contains a considerable literature, more than other local African languages, (5) there are several methodological grammar and phonetics studies and (6) several translations from languages such as English and (7) it is spoken by old and young alike.

There has been an interest in developing digital tools for language documentation and natural language processing. Such initiatives have come from researchers involved in linguistics, such as those belonging to linguistics departments at universities in Malawi and Zambia. For example, in Malawi, we found the Chichewa monolingual dictionary corpus containing about 13,000 nouns or this one phonetically annotated short corpus.

The comparative online Bantu dictionary at Berkley includes a dataset for Chichewa, however, the project seems to have stalled in 1997. More recently, there has been an interest in creating datasets used in NLP tools and machine translation and, recently, according to Professor Kishindo, there is a PhD candidate at the University of Malawi interested in working on Machine Translation for Chichewa.

From our investigation, we observe that these datasets or tools tend to be kept in the private domain, are not regularly maintained, or are used only once, and are not well documented. However, their existence is important and it shows that there is a desire and need for such tools.

Conclusions

Chichewa is an important African language. There are differences between the main dialects of Chichewa and the language is undergoing continuous change. Improved methods for discovering online content and digitizing text can open new opportunities for organising Chichewa text into useful corpora. These can then be useful in linguistic work, in building tools for manipulating and comparing text, for finding and visualising connections between texts and for improving machine translation.

Chichewa continues to change as new terms are added to the vocabulary arising from technological needs for example. Its use by the younger generation creates new idioms and meaning, and the creative expressions through poetry and literature find venues online. Looking at language in new and novel ways using technology, can also help engage with the new generation in how they use, view and develop their language.

In this short article, we looked at the use of Chichewa and why we think it is important to build data sets for this language. We hope that this will be motivating and inspiring to others who are interested in this language or other African languages. This article was written as the author embarked on an AI4D Language Dataset Fellowship for putting together a Chichewa dataset. This is a small but important initiative aimed at engaging with the Machine Learning generation on the African continent. I am honoured to be a small part in the building of such datasets.

Researcher Profile: Amelia Taylor

Amelia graduated with a PhD in Mathematical Logic from Heriot-Watt University in 2006 where I was part of the ULTRA group. After that she worked as a research assistant on a project with Heriot-Watt University and the Royal Observatory in Edinburgh, aiming at developing an intelligent query language for astronomical data. From 2006 to 2013, Amelia also worked in finance in the City of London and Edinburgh – she built risk models for asset allocation and liability-driven investments. F

or the last 5 years, Amelia has been teaching programming and AI courses at the University of Malawi in the CIT and engineering department. Amelia also teaches research methodology and supervises MSc and PhD students. While my first interest in AI as an undergraduate was in the field of Natural Language Processing and intelligent query systems, she is interested in the other use of technology and AI for solving real-world problems.

Partners

Disclaimer

Knowledge 4 All Foundation Ltd.