(RU) most popular ML algorithms explained in simple terms

vas3k.ru/blog/machine_learning/

#data_science

Машинное обучение для людей

Разбираемся простыми словами

DS/ML digest 24

Key topics of this one:

- New method to calculate phrase/n-gram/sentence embeddings for rare and OOV words;

- So many releases from Google;

spark-in.me/post/2018_ds_ml_digest_24

If you like our digests, you can support the channel via:

- Sharing / reposting;

- Giving an article a decent comment / a thumbs-up;

- Buying me a coffee (links on the digest);

#digest

#deep_learning

#data_science

2018 DS/ML digest 24

2018 DS/ML digest 24 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me

Using sklearn pairwise cosine similarity

scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.html#sklearn.metrics.pairwise.cosine_similarity

On 7k * 7k example with 300-dimensional vectors it turned out to be MUCH faster than doing the same:

- In 10 processes;

- Using numba;

The more you know.

If you have used it - please PM me.

#nlp

Understanding the current SOTA NMT / NLP model - transformer

A list of articles that really help to do so:

- Understanding attention jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/

- Annotated transformer nlp.seas.harvard.edu/2018/04/03/attention.html

- Illustrated transformer jalammar.github.io/illustrated-transformer/

Playing with transformer in practice

This repo turned out to be really helpful

github.com/huggingface/pytorch-openai-transformer-lm

It features:

- Decent well encapsulated model and loss;

- Several head for different tasks;

- It works;

- Ofc their data-loading scheme is crappy and over-engineered;

My impressions on actually training the transformer model for classification:

- It works;

- It is high capacity;

- Inference time is ~`5x` higher than char-level or plain RNNs;

- It serves as a classifier as well as an LM;

- Capacity is enough to tackle most challenging tasks;

- It can be deployed on CPU for small texts (!);

- On smaller tasks there is no clear difference between plain RNNs and Transformer;

#nlp

Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)

May 25th update: New graphics (RNN animation, word embedding graph), color coding, elaborated on the final attention example. Note: The animations below are videos. Touch or hover on them (if you’re using a mouse) to get play controls so you can pause if needed. Sequence-to-sequence models are deep learning models that have achieved a lot of success in tasks like machine translation, text summarization, and image captioning. Google Translate started using such a model in production in late 2016. These models are explained in the two pioneering papers (Sutskever et al., 2014, Cho et al., 2014). I found, however, that understanding the model well enough to implement it requires unraveling a series of concepts that build on top of each other. I thought that a bunch of these ideas would be more accessible if expressed visually. That’s what I aim to do in this post. You’ll need some previous understanding of deep learning to get through this post. I hope it can be a useful companion to reading the papers mentioned…

Gensim's fast-text subwords

Some monkey patching to get subwords from Gensim's fast-text

from gensim.models.utils_any2vec import _compute_ngrams,_ft_hash

def subword(self, word):

ngram_lst = []

ngrams = _compute_ngrams(word, self.min_n, self.max_n)

for ngram in ngrams:

ngram_hash = _ft_hash(ngram) % self.bucket

if ngram_hash in self.hash2index:

ngram_lst.append(ngram)

return ngram_lst

gensim.models.keyedvectors.FastTextKeyedVectors.subword = subword

Useful Python / PyTorch bits

dot.notation access to dictionary attributes

class dotdict(dict):

__getattr__ = dict.get

__setattr__ = dict.__setitem__

__delattr__ = dict.__delitem__

PyTorch embedding layer - ignore padding

nn.Embedding has a padding_idx attribute not to update the padding token embedding.

#python

#pytorch

youtu.be/HvH0b9K_Iro

This AI Performs Super Resolution in Less Than a Second

SeNet

- arxiv.org/abs/1709.01507;

- A 2017 Imagenet winner;

- Mostly ResNet-152 inspired network;

- Transfers well (ResNet);

- Squeeze and Excitation (SE) block, that adaptively recalibratess channel-wise feature responses by explicitly modelling in- terdependencies between channels;

- Intuitively looks like - convolution meet the attention mechanism;

- SE block:

- pics.spark-in.me/upload/aa50a2559f56faf705ad6639ac973a38.jpg

- Reduction ratio r to be 16 in all experiments;

- Results:

- pics.spark-in.me/upload/db2c98330744a6fd4dab17259d5f9d14.jpg

#deep_learning

Chainer - a predecessor of PyTorch

Looks like

- PyTorch was based not only on Torch, but also its autograd was forked from Chainer;

- Chainer looks like PyTorch ... but not by Facebook, but by independent Japanese group;

- A quick glance through the docs confirms that PyTorch and Chainer APIs look 90% identical (both numpy inspired, but using different back-ends);

- Open Images 2nd place was taken by people using Chainer with 512 GPUs;

- I have yet to confirm myself that PyTorch can work with a cluster (but other people have done it) github.com/eladhoffer/convNet.pytorch;

www.reddit.com/r/MachineLearning/comments/7lb5n1/d_chainer_vs_pytorch/

docs.chainer.org/en/stable/comparison.html

#deep_learning

eladhoffer/convNet.pytorch

ConvNet training using pytorch. Contribute to eladhoffer/convNet.pytorch development by creating an account on GitHub.

Also - thanks for all DO referral link supporters - now finally hosting of my website is free (at least for next ~6 months)!

Also today I published a 200th post on spark-in.me. Ofc not all of these are proper long articles, but nevertheless it's cool.

DS/ML digest 23

The key topic of this one - is this is insanity

- vid2vid

- unsupervised NMT

spark-in.me/post/2018_ds_ml_digest_23

If you like our digests, you can support the channel via:

- Sharing / reposting;

- Giving an article a decent comment / a thumbs-up;

- Buying me a coffee (links on the digest);

Let's spread the right DS/ML ideas together.

#digest

#deep_learning

#data_science

2018 DS/ML digest 23

2018 DS/ML digest 23 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me

MySQL - replacing window functions

Older versions of MySQL (and maybe newer ones) do not have all the goodness you can find in PostgreSQL. Ofc you can do plain session matching in Python, but sometimes you just need to do it in plain SQL.

In Postgres you usually use window functions for this purpose if you need PLAIN SQL (ofc there are stored procedures / views / mat views etc).

In MySQL it can be elegantly solved like this:

SET @session_number = 0, @last_uid = '0', @current_id = '0', @dif=0;

SELECT

t1.some_field,

t2.some_field,

...

@last_uid:[email protected]_uid,

@current_uid:=t1.uid,

@dif:=TIMESTAMPDIFF(MINUTE, t2.session_ts, t1.session_ts),

if(@[email protected]_uid, if(@dif > 30,@session_number:[email protected]_number+1,@session_number),@session_number:=0) as session

FROM

table1 t1

JOIN table2 t2 on t1.id = t2.id+1

#data_science

Training a MNASNET from scratch ... and failing

As a small side hobby we tried training new Google's mobile network from scratch and failed:

- spark-in.me/post/mnasnet-fail-alas

- github.com/snakers4/mnasnet-pytorch

Maybe you know how to train it properly?

Also now you can upvote articles on spark in me! =)

#deep_learning

Training your own MNASNET

Training your own MNASNET Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me

youtu.be/cEBgi6QYDhQ

Everybody dance now! - AI-Based Motion Transfer

A small hack to spare PyTorch memory when resuming training

When you resume from a checkpoint, consider adding this to save GPU memory:

del checkpoint

torch.cuda.empty_cache()

#deep_learning

DS/ML digest 22

spark-in.me/post/2018_ds_ml_digest_22

#digest

#deep_learning

#data_science

2018 DS/ML digest 22

2018 DS/ML digest 22 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me

ADAMW to be integrated into upstream PyTorch?

github.com/pytorch/pytorch/pull/3740

#deep_learning

Fixing Weight Decay Regularization in Adam by jingweiz · Pull Request #3740 · pytorch/pytorch

Hey, We added SGDW and AdamW in optim, accoridng to the new ICLR submission from Loshchilov and Hutter: Fixing Weight Decay Regularization in Adam. We also found some inconsistency of the current i...

Crowd-AI maps repo

Just opened my repo for crowd AI maps 2018.

Did not pursue this competition till the end, so it is not polished, .md is not updated. Use it at your own risk!

github.com/snakers4/crowdai-maps-2018

spark-in.me/post/a-small-case-for-search-of-structure-within-your-data

#deep_learning

snakers4/crowdai-maps-2018

CrowdAI mapping challenge 2018 solution. Contribute to snakers4/crowdai-maps-2018 development by creating an account on GitHub.

Venn diagrams in python

Compare sets as easy as:

# Import the library

import matplotlib.pyplot as plt

from matplotlib_venn import venn3

# Make the diagram

plt.figure(figsize=(10,10))

venn3(subsets = (s1,s2,s3),set_labels=['synonyms','our_tree','add_syns'])

plt.show()

Very simple and useful.

#data_science

Garbage collection and encapsulation in python

Usually this is not an issue.

But when doing a batch-job / loop / something with 100k+ objects it suddenly becomes an issue.

Also it does not help that the environment you are testing your code and the actual running environments are different.

The solution - simply encapsulate everything you can. Garbage is automatically collected.

#data_science

A case against Kaggle

If you thought that Kaggle is the home of Data Science - think again.

www.kaggle.com/c/airbus-ship-detection/discussion/64355

This is official - they do not know the hell they are doing.

There have been several appalling cases already, but this takes the prize.

Following this thread, wrote a small petition to Kaggle

www.kaggle.com/c/airbus-ship-detection/discussion/64393

I doubt that they will hear, but why not.

#data_science

A small bug in PyTorch to numpy conversions

Well, maybe a feature =)

When you do something like this

a.permute(0, 2, 3, 1).numpy()

or

a.view((same shape here)).numpy()

you may expect behaviour similar to np.reshape.

But no. Looks like now all of these functions are implemented in C and they produce artifacts. To avoid artifacts call:

a.permute(0, 2, 3, 1).contiguous().numpy()

a.view((same shape here)).contiguous().numpy()

It is also iteresting, because when you apply some layers in PyTorch error is raised when you do not use .contiguous().

#deep_learning

Played with FAISS

Played with FAISS on the GPU a bit. Their docs also cover simplistic cases really well, but more sophisticated cases are better inferred from examples, because their Python docs are not really intended for heavy use.

Anyway I manged to build a KNN graph with FAISS on on GPU for 10m points in 2-3 hours.

It does the following:

- KNN graph;

- PCA, K-Means;

- Queries;

- VERY sophisticated indexing with many option;

It supports:

- GPU;

- Multi-GPU (I had to use env. variables to limit GPU list, because there is no option for python);

Also their docs are awsome for such a low-level project.

github.com/facebookresearch/faiss/wiki/Faiss-building-blocks:-clustering,-PCA,-quantization

#data_science

#similarity

facebookresearch/faiss

A library for efficient similarity search and clustering of dense vectors. - facebookresearch/faiss

Playing with Atom and Hydrogen

TLDR - it delivers to turn the Atom editor into something like interactive notebook, but its auto-completion options are very scarce. Also you can connect to your running ipython kernel as well as docker container.

nteract.gitbooks.io/hydrogen/docs/Installation.html

blog.nteract.io/hydrogen-interactive-computing-in-atom-89d291bcc4dd

But it is no match to either a normal notebook or a python IDE.

#data_science

youtu.be/P0fMwA3X5KI

NVIDIA's Image Restoration AI: Almost Perfect

This is madness

Probably impossible to replicate

Also code is available for PyTorch =)

2018 DS/ML digest 21

spark-in.me/post/2018_ds_ml_digest_21

#digest

#deep_learning

#nlp

2018 DS/ML digest 21

2018 DS/ML digest 21 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me

PyTorch - a brief state of sparse operations

TLDR - they are not there yet

- github.com/pytorch/pytorch/issues/9674

If you would like to see them implemented faster, write here

- github.com/pytorch/pytorch/issues/10043

#deep_learning

The state of sparse Tensors #9674

This note tries to summarize the current state of sparse tensor in pytorch. It describes important invariance and properties of sparse tensor, and various things need to be fixed (e.g. empty sparse tensor). It also shows some details of ...

Nice down-to-earth post about Titan V vs 1080 Ti

medium.com/@u39kun/titan-v-vs-1080-ti-head-to-head-battle-of-the-best-desktop-gpus-on-cnns-d55a19866b7c

Also it seems that new generation nvidia GPUs will have 12 GB of RAM tops.

1080Ti is the best option for CNNs now.

#deep_learning

Titan V vs 1080 Ti — Head-to-head battle of the best desktop GPUs on CNNs. Is Titan V worth it? 110 TFLOPS! no brainer, right?

NVIDIA’s Titan V is the latest “desktop” GPU built upon the Volta architecture boasting 110 “deep learning” TFLOPS in the spec sheet. That…

Found all Ipython's rich display capabilities in one place

nbviewer.jupyter.org/github/ipython/ipython/blob/1.x/examples/notebooks/Part%205%20-%20Rich%20Display%20System.ipynb#Links-to-local-files

Notebook on nbviewer

Check out this Jupyter notebook!

Google updates its transformer

ai.googleblog.com/2018/08/moving-beyond-translation-with.html

#nlp

Moving Beyond Translation with the Universal Transformer

Posted by Stephan Gouws, Research Scientist, Google Brain Team and Mostafa Dehghani, University of Amsterdam PhD student and Google Research...

youtu.be/LBezOcnNJ68

NVIDIA's AI Makes Amazing Slow-Mo Videos

Yet another crowd GPU rent service?

vast.ai/console/create/

Create Instance | Vast.ai Console

Search available instances, configure launch settings, create instances