A researcher is doing research on a certain topic, e.g. sentiment extraction in the domain of political conflicts. She wants to find all relevant literature and datasets that she could use for her research and/or construction of an application. Publications that might be of interest to her are stored at various points, e.g. OpenAire but also institutional/university repositories, various publishers' sites (to which she may have access through academic subscriptions but not necessarily), research repositories etc. Datasets, i.e. corpora (raw & annotated) as well as lexica, vocabularies, ontologies etc., are also scattered at various repos & portals. They are described with different metadata schemas, depending on the resource type & repository. She would ideally like to make one query at one point and get a list of all resources of relevance, a common description and a quick gist of the contents of the resource (e.g. abstract of the publication, sample of the dataset).

It focuses on the core issue of the WG’s interests, i.e. resource discovery & interoperability problems between different resource types and different metadata schemas used to describe them. Authentication/authorization depending on the user profile is also required.

They have come across similar situations and know how important description is for resource discovery, especially when it involves different resource types.

By standardizing metadata schemas and mappings between the most popular ones, at least; addressing the way similar information (e.g. classification) is encoded in different resource types and communities is of great importance.

External experts in this WG work in various areas where resources of different types are described with various metadata schemas. Their expertise will be of value for finding a "common/standard language" between them.

The EC and other funders are looking into ways to discover correlations and trends in funded research in Europe. They want to go beyond the obvious relationships (project-publication-data-patents-software) and the related statistics (i.e., who produces what and co-funds). They need to identify the hidden relationships brought out via concepts and topics as these are discovered through advanced NLP and probabilistic algorithms. They need to find various side effects, e.g. structural effects stemming from collaboration from different institutions, countries and domains (cross-discipline)

As an addition, EC wants to compare their programmes with the ones in the US (NSF, NIH).

To get some meaningful results the following tasks need to be completed:

Relevant to our research analytics experts. Also practitioners that will deal more with structured data and face semantic interoperability problems with different semantic classification schemes.

Discovery of software solutions with appropriate software versioning and dependencies is a common problem to many TDM users (and developers alike!).

Promotion of standard metadata-based descriptions of software, standard (or standardisation of) vocabularies/authority lists and semantic interoperability of classification schemes & promotion (or standardisation) of licensing schemes for ensuring are among the main objectives.

Systems/methods/schemes used for metadata-based discovery. Vocabularies used.

An open access journal or a repository wants to enrich the metadata of its content. Europe and national data services are funding infrastructures/aggregators like OpenAIRE and CORE to act as such information hubs/brokers. To be more specific, OpenAIRE operates a set of TDM tools on the aggregated content from 15mi publications and related data. These tools (either operated on the metadata+abstract or the full text) are able to extract all research related information (grants, authors and their affiliations, data/software/patent citations, classifications, topics, etc.) and clean/enrich the harvested metadata records. In addition, via de-duplication and disambiguation mechanisms OpenAIRE is able to link all results, organizations, authors, funders and data into an integrated research information system.

This scenario can be used in the following ways:

Automatic metadata extraction of content is a common requirement in many repositories. Mechanisms for documenting particular mining services discoverable and integratable/callable from within such e-infrastructures are required.

Through promoting guidelines and schema(s) for formal metadata-based documentation and providing mechanisms for integrating mining services.

What schemas do experts and practitioners use? Level of detail and completeness of metadata elements. Use of predefined vocabularies and value sets. Protocols for harvesting and exchanging research information.

A user likes to make a detailed comparison of multiple versions of a language resource or a knowledge base. The user is interested in running a TDM experiment using a resource (e.g. Wikidata) and likes to evaluate the performance changes of the experiment with regards to using different versions of the same resource. To enable an analytical study, he requires detailed statistics related to revision changes (e.g. total number of concepts, total number of relations, number of added/updated/deleted concepts/relations). However, the problem is that this type of information is not typically shipped with resources.

Knowledge encoded in a resource often has a great impact on results of an experiment; hence, they affect the reproducibility of results. However, statistical information regarding a resource or different revisions of a resource usually are not either well documented or published with the resource itself. This makes it difficult to trace changes made to a resource and also makes it difficult for researchers to analyze their results or compare with previous results.

Providing detailed information about a resource makes it more transparent and easier to be adopted by the community. It also makes it easier to compare multiple resources prior to using them.

Through designing metadata placeholder for statistical metadata of resources and implementing mechanism to calculate data that is not inherently announced with the resource.

It is required to identify which metadata types are suited for describing each resource. The other requirement is to create components to generate statistical metadata information for each resource.

A novice researcher, who may be in a research organization setting, an SME, or a different type of organization like a museum or a ministry that does not have access to a library subscription, wants to get some insights on his/her topic of interest. For example, an archaeologist in the Acropolis Museum trying to find scientific evidence on how to use storytelling mechanisms in a museum setting. This is a very trendy and state of the art technology and she would like specifically to know about which museums have applied the specific technologies on what types of settings, and moreover retrieve the ones that have somehow led to products. She will use this output within a research project which will ultimately help facilitate an iphone/android app for mobiles or tablets, which will be owned by the Museum and will be available for free download from the Apple/GPlay stores. The researcher does not know where to look and has no knowledge of the text mining algorithms or tools. She is acquainted to newer technologies as she has a MS. in cultural heritage digital technologies.

Explore which content to search into, which probably should include different content types: scientific publications, technical and project reports. She needs an easy way to discover what is around and make the appropriate choices. The researcher needs to find the TDM services (not tools, because they must be easy to use and the researcher has no own computing resources) that will do the job for her. She needs to identify them in a straightforward manner and an easy way to point to the content to be mined. It is also important to have the results in a human readable way. All content and services need to be run in the cloud as the museum doesn’t have any supporting digital infrastructure. Authentication/authorization is also an important aspect that needs to be taken into consideration.

External experts will want to know how the specific content to be mined can be assembled into a virtual collection from different content types, and how text mining tools/workflows can be discovered.

How are fairly advanced applications (e.g. storytelling) technologies formally described or potentially composed so that they are discoverable by non-expert users?

The user wants to build a knowledge base on animal diseases using information extracted from scientific or technical documents. The domain model is represented in an ontology describing relations between animals (hosts), diseases and pathogens (vectors). The lexical elements to identify the different objects are spread in disconnected resources (e.g. list of “livestock” from Agrovoc, “diseases” from a custom lightly hierarchical vocabulary and “pathogens” from the Uniprot taxonomy). Those resources provide information for scientific and common names, abbreviations, etc. There can be several relevant for a same type of object.

WG2 specific tasks:

This scenario requires uniform access to content from heterogeneous resources. In order to operationalise these resources, we need interoperability of their content enabling uniform access.

The user has to deal with vocabularies presenting different schemas and levels of information. This requires schema mapping. The vocabulary elements of these resources can be of many forms, e.g.

Detecting these elements in text may require different strategies or tools as they contain Named Entities (form and case sensitivity) or terms (fuzzy recognition, case insensitivity, etc.).

Are vocabularies really reusable and combinable…​ and if yes, to what extent?

A user wants to analyze mentions of pharmaceutical products based on the main agents used in these products. The user uses a publicly accessible dataset of products and their agents, but the dataset is incomplete, e.g. does not include products still in testing stage. The user wants to provide additional (confidential/unlicensed/proprietary) data about product/agents that should be taken into account for the analysis.

WG2 specific tasks:

Use-case involves a public and a custom/private knowledge source

Including custom content and/or confidential data is a common problem in infrastructures

The general idea is to treat keywords attached to a publication as a "mini lexicon" representing its topic and expand/enrich this lexicon by combining it with lexical/conceptual resources such as domain lexica, glossaries and ontologies. Possible applications of this idea:

WG2 specific tasks: KB content harmonization including semantic mapping and multilingual keyword expansion.

It involves the interoperability between various language resources (publications, lexical resources, ontologies etc.).

Interoperability issues between resources of the same or different type (e.g. lexica, keywords, term banks) is a central topic. Combining this with document search extends resource coverage

Keywords should be in some way qualified with the source of the keyword if it belongs to a controlled vocabulary. E.g. a keyword could be a URI or there could be an additional metadata field indicating where keywords come from (latter related to WG1).

Researcher A is conducting research on a variety of sources. He needs to extract statistical information from 3 DB: X, Y and Z. He runs a TDM instance on the 3 databases and obtains insightful results. He wants to know how, if at all, he can use the resulting output. Database X is made publicly available on the Internet by the right holder but is not released under any specifically identified licence, term of use or any other policy. DB X is constituted by a text corpus (i.e. the individual elements constituting the database are not individually copyrightable). Database Y is available online under a CC-BY-SA 4.0 license and is constituted by a DB of academic articles (i.e. the elements constituting the database are individually copyrightable). Database Z is a “closed” database to which the researcher has access thanks to the “academic subscription” of its institution which allows to TDM the database but only for “academic and research activities and for non commercial purposes” and forbids the distribution or communication of results for any purpose. All these activities are performed using tools and or services belonging to the researcher (i.e. no licence or terms of use on tools and or services are relevant in this scenario).

The scenario identifies the main elements of legal interoperability: absence of clear license or right statement, OA publishing, closed databases, Exceptions and Limitations to Copyright. The scenario also considers the type of results that TDM activities lead to (whether there is a redistribution of at least part of the original data).

External experts combine expertise in the legal and technical field. The combination of the three identified DB legal conditions is the starting point to determine legal interoperability.

The same scenario as scenario 1 above applies, however database (Y) is not directly accessible. On the contrary, the database owner allows the user (on the basis of the academic subscription) to run the rights holder dedicated TDM web service. Basically, the researcher does not have direct access to the database, but he is only able to place a query and obtain a result. The TDM web service is made available to the researcher for “non commercial uses” only. On the other hand, database Z is directly accessed but using the software (freely downloadable) of Company Alfa. The licence of software Alfa allows users to use the software only for non commercial purposes. For the purpose of this scenario, database Y and Z are not protected by any IP rights nor their use is contractually restricted, i.e. they are in the public domain (i.e. the scope of the question is to test the enforceability of services’ and tools’ agreements in relation to the results of the TDM activity).

The results of the TDM so obtained are all combined together in one new DB. Can the researcher publish a paper based on those results? Can he publish the paper under a CC-BY 4.0 license. Can he also publish the DB itself under a CC-BY 4.0 license? If yes, is he bound to apply a NC licence? Can the researcher develop a commercial product and sell it? Would the fact that the “results” reproduce substantial or insubstantial parts of the original DB, or on the contrary that those results do not reproduce parts of the original DB (for example the statistical recurrence of a given word in the DBs, but not the word itself) make a difference in the result?

The scenario adds a main element of complexity: some TDM services restrict the use that you can do with the results obtained from their services. In this case researchers are not directly accessing a database but they are using a service offered by the data provider to analyse its own DB. Of particular relevance when the DB is not protected by copyright or sui generis database rights (SGDR ) and therefore the SLA (service level agreement) is the only basis to restrict use of the results (not backed-up by property rights).

External experts combine expertise in the legal and technical field. The combination of the DB legal conditions and the add-on of the service element an additional step toward the determination of legal interoperability.

The same scenario as scenario 1 above applies. Would it make any difference if the resulting DB, although being based on the data of the original databases, does not reproduce any substantial or insubstantial parts of those databases? In other words, the results of TDM are directly based on the analysis of the original DBs but do not materially reproduce any part of them? For example the results of the TDM activity is "in the analysed corpora the word "and" occurs 235 times".

The scenario looks into the scope of copyright and SGDR protection. To what extent a DB that "derives" from another one, without however reproducing it, can be considered a derivative of the originals, under both copyright rules and SGDR rules?

External experts combine expertise in the legal and technical field. Try to identify the limits in the scope of the applicable legal tools is a fundamental element.

I am building a text mining workflow in a specific workflow editor (e.g. Argo). There is a really useful component, but it is in a repository for a different workflow editor (say GATE). I cannot use this really useful Gate workflow (which may be made of one or more modules) in my Argo workflow. I must either reimplement the really useful component in Argo or convert the work I have already done into Gate.

This scenario exposes the core issue in workflow interoperability. Because different workflow editors use differing annotation schema, a component written for one workflow ties the user into that specific ecosystem. This has a direct negative impact on the scientific goals of openness and reproducibility.

External experts will want a smooth experience when using a TDM framework. Having to reimplement components is frustrating and costs time and money.

I want to compare competing components for TDM (e.g. which is the best parser/tokeniser/event extraction component for my text?). I have a workflow which is made up of several components (e.g. A tokeniser, A POS tagger and an event extractor). I have several options for each component. I would like to know if changing the tokenizer / POS tagger / event extractor will improve or degrade my results. I would like to know if changing a parameter of the tokenizer / POS tagger, etc. (e.g. which model to use, which kinds of abbreviation lists to use, etc.) affects the results. I would like to know if a specific version of a component which I use will improve my results. This not only be the version “1.2.3” but be the “latest”, or “latest stable”, or “latest snapshot” and the respective version is automatically looked up in a repository.

Workflows provide a seamless way to replace components with similar functionality. In the easiest case, competing components will have exactly the same inputs and outputs as each other and the user will be able to directly replace one component with another. In a more complicated scenario components may require extra resources or annotations. These can still be incorporated, as long as the prerequisite components are also available.

External experts will be interested to know which option from many components is the best for their specific task. If it is difficult to interface components (say, a wrapper must be written for each new component), then the expert is likely to only evaluate one or two components. The more components they can compare for a specific sub-task, the likelier it becomes that they will find the correct component for their purposes.

By creating specification for interoperable components, OpenMinTeD will provide a framework for the easy comparison of components within a workflow setting.

Types of components where they would like to compare competing workflows. Experiences of building workflows, but not knowing which tagger / tokenizer / etc. to use or how to parametrize it.

I want to use text mining techniques to analyse literature in an area that has not previously been examined with TDM (e.g. paleontology). Unfortunately, although I am an expert in the subject area, I know very little about language processing or TDM. I want to start with annotating a set of documents. How do I retrieve the articles? Where do I find the necessary tools? Do I need dedicated hardware or infrastructure? Where should I start?

This scenario, taking a non-expert perspective, shows how hard it is to start TDM research, even by building the simplest workflow. Most of them require a user to prepare infrastructure, install and configure the tools, and convert the data into a specified format. Only a small fraction of TDM tools is available as simple-to-use web tools.

External experts frequently have little knowledge about TDM but possess domain expertise necessary to create annotated resources. They know how the TDM field looks “from the outside”.

OpenMinTeD could build (or at least help to build by providing interoperability standards) a single, easy to use interface that would let inexperienced users to employ different tools in their work, regardless of their data formats and infrastructure requirements. What is more, OpenMinTeD could provide a simple guide that will help them to choose the right text representation standard, metadata format and license.

Experiences of trying to build resources without expertise in the field: which was the hardest step? What kind of information they needed?

I want to give readers of my paper the possibility to re-run my experiments involving text processing with modified settings or data and check the results. Unfortunately, contrary to statistical data analysis, it is not enough to provide the input files and program script — instead, the reader would have to install all the tools I used in the project in his computing environment, configure them, take care for data format conversion, etc. As he is very unlikely to be able to do it, my results are virtually irreproducible.

The problem in this scenario is that although an interested paper reader may be a TDM expert, he/she probably uses different workflow and environment. To make him able to analyse a given data set, we need the possibility to run a given program without the user manually installing and configuring all the tools on a target environment (e.g. a local PC, a local server, a cloud server, etc…). The user may either choose to re-run the experiment with slightly different parameters or input data, or he may edit the workflow (e.g. replace some of the components) and check whether the conclusions remain valid. That clearly requires a lot of work in the area of annotations and workflows interoperability.

Here, experts with experiences in TDM would be very helpful, because they know what prevents them from sharing all their input data and procedures or replicating workflows of others.

OpenMinTeD could build (or at least help to build by providing interoperability standards) a single, easy to use interface that would let curious readers repeat the experiments without cost and time needed to fully install the whole environment. In case it is not possible, OpenMinTed could alternatively provide standards, best practices, and tooling to facilitate the installation of the environment. The latter is also an important step towards dynamically deploying TDM workflows to computing resources in the age of scalable cloud computing.

They could share knowledge about what prevents them from sharing input and procedures of their research projects. Is it licensing? Or maybe amount of work necessary to fully describe the computing environment?

I, project manager, need a TDM workflow in order to provide TM/NLP functionality to the service we are developing. Examples:

The developers need to know how to invoke the workflow when new input is provided and how to fetch the result of the TDM analysis. They might require continuous processing of incoming text.

They will also require some level of robustness, the TDM should then be able to provide failure/recovery information. Also they might want to get as many results as possible as soon as possible even if the whole workflow has not finished.

From the workflow design PoV, we assume in this scenario that the team includes one TDM professional. However they might want to optimize the components according to specific criteria (storage, CPU usage, robustness, scalability).

Experience of deploying enterprise or end-user services that includes some level of TDM would be welcome.

OpenMinTeD could specify a standard API to access workflows (parameters, LR), output (annotations, clusters, classifiers), and status (running, fail).

From experts:

From WGs:

A TDM specialist or a domain expert is building a linguistic/semantic resource, it could be:

These resources are typically used in other workflows. However the development of resources itself requires a TDM workflow. On one hand it could feed the user with suggestions (pre-annotation for corpus annotation, IE or term extraction for terminology design). On the other hand the evaluation of the resource can be partially achieved through a TDM workflow (inter-annotator agreement, testing with the resource).

The TDM workflow could have protocols compatible with the main annotation and ontology editors on the market.

Moreover the manual intervention in the middle of the workflow poses a challenge because it cannot be designed exclusively as a batch process. The pace of human produced output is at the same time slower and more valuable than the result of fully automated output. The TDM workflow could proceed either continuously as the experts modify the resource, or with partial input.

Experience and insights on not strictly batch workflows.

Allow interactive steps within the TDM workflow. Propose solutions compatible with editors.

Knowledge about how they have been solving the problem so far and how could OpenMinTeD improve the process.