Spark in me - Internet, data science, math, deep learning, philo

snakers4 @ telegram, 1361 members, 1660 posts since 2016

All this - lost like tears in rain.

Data science, deep learning, sometimes a bit of philosophy and math. No bs.

Our website
- spark-in.me
Our chat
- goo.gl/WRm93d
DS courses review
- goo.gl/5VGU5A
- goo.gl/YzVUKf

Posts by tag «deep_learning»:

snakers4 (Alexander), December 19, 08:16

DS/ML digest 32

Highlights:

- A way to replace softmax in NMT;

- Large visual reasoning dataset;

- PyText;

spark-in.me/post/2018_ds_ml_digest_32

#digest

#deep_learning

#data_science

2018 DS/ML digest 32

2018 DS/ML digest 32 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), December 17, 09:24

PyText

- PyText github.com/facebookresearch/pytext from Facebook:

- TLDR - FastText meets PyTorch;

- Very similar to AllenNLP in nature;

- Will be useful if you can afford to write modules for their framework to solve 100 identical tasks (i.e. like Facebook with 200 languages);

- In itself - seems to be too high maintenance to use;

I will not use use it.

#nlp

#deep_learning

facebookresearch/pytext

A natural language modeling framework based on PyTorch - facebookresearch/pytext


snakers4 (Alexander), December 09, 07:59

DS/ML digest 31

Highlights of the week:

- PyTorch 1.0 released;

- Drawing with GANs;

- BERT explained;

spark-in.me/post/2018_ds_ml_digest_31

#digest

#deep_learning

#data_science

2018 DS/ML digest 31

2018 DS/ML digest 31 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), December 08, 05:17

PyTorch 1.0 release

View Release:

 github.com/pytorch/pytorch/releases/tag/v1.0.0

#deep_learning

pytorch/pytorch

Tensors and Dynamic neural networks in Python with strong GPU acceleration - pytorch/pytorch


snakers4 (Alexander), December 06, 09:04

Painting with GANs

This looks just awesome.

I guess it will not work in real resolutions yet.

gandissect.res.ibm.com/ganpaint.html?project=churchoutdoor&layer=layer4

#deep_learning

Painting with GANs from MIT-IBM Watson AI Lab

This demo lets you modify a selection of meaningful GAN units for a generated image by simply painting.


snakers4 (Alexander), November 29, 08:10

Article about the reality of CV in Russia / CIS

(RU)

cv-blog.ru/?p=253

Also a bit on how to handle various types of "customers", who want to contract CV systems from you.

Warning - too much harsh reality)

#deep_learning

snakers4 (Alexander), November 28, 11:55

DS/ML digest 30

spark-in.me/post/2018_ds_ml_digest_30

#digest

#deep_learning

#data_science

2018 DS/ML digest 30

2018 DS/ML digest 30 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), November 22, 11:56

Our victory in CFT-2018 competition

TLDR

- Multi-task learning + seq2seq models rule;

- The domain seems to be easy, but it is not;

- You can also build a pipeline based on manual features, but it will not be task agnostic;

- Loss weighting is crucial for such tasks;

- Transformer trains 10x longer;

spark-in.me/post/cft-spelling-2018

#nlp

#deep_learning

#data_science

Winning a CFT 2018 spelling correction competition

Building a task-agnostic seq2seq pipeline on a challenging domain Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), November 22, 11:49

Problems with GPUs in DL box

Cryptic messages like:

GPU is lost

Usually this is either:

- PSU;

- Or bad PCIE contact;

- Too much load on PCIE bus;

#deep_learning

snakers4 (Alexander), November 15, 08:09

DS/ML digest 29

spark-in.me/post/2018_ds_ml_digest_29

#digest

#deep_learning

#data_science

2018 DS/ML digest 29

2018 DS/ML digest 29 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), November 06, 13:45

DS/ML digest 28

Google open sources pre-trained BERT ... with 102 languages ...

spark-in.me/post/2018_ds_ml_digest_28

#digest

#deep_learning

#data_science

2018 DS/ML digest 28

2018 DS/ML digest 28 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), November 06, 12:38

A small saga about keeping GPUs cool

(1) 1-2 GPUs with blower fans (or turbo fans) in a full tower

-- idle 40-45C

-- full load - 80-85C

(2) 3-4 GPUs with blower fans (or turbo fans) in a full tower

-- idle - 45-55C

-- full load - 85-95С

Also with 3-4+ GPUs your room starts to heat up significantly + even without full fan speed / overclocking the sound is not very pleasant.

Solutions:

(0) Add a corrugated air duct to dump heat outside minus 3-5C under load;

(1) Add a high-pressure fan to blow between the GPUs minus 3-5C under load;

(2) Place the tower on the balcony minus 3-5C under load;

In the end it is possible to achieve <75C under full load on 4 or even 6 GPUs.

#deep_learning

snakers4 (Alexander), November 03, 09:40

Building client routing / semantic search and clustering arbitrary external corpuses at Profi.ru

A brief executive summary about what we achieved at Profi.ru.

If you have similar experience or have anything similar to share - please do not hesitate to contact me.

Also we are planning to extend this article into a small series, if it gains momentum. So please like / share the article if you like it.

spark-in.me/post/profi-ru-semantic-search-project

#nlp

#data_science

#deep_learning

Building client routing / semantic search and clustering arbitrary external corpuses at Profi.ru

Building client routing / semantic search and clustering arbitrary external corpuses at Profi.ru Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), October 27, 10:16

Canonical one-hot encoding one-liner in PyTorch

Or 2 liner, whatever)

# src - is the input tensor (batch,indexes)

trg_oh = torch.FloatTensor(src.size(0), src.size(1), self.tgt_vocab).zero_().to(self.device)

trg_oh.scatter_(2, trg, 1)#deep_learning

snakers4 (Alexander), October 24, 09:11

Concurrent Spatial and Channel Squeeze &amp; Excitation in Fully Convolutional Networks

- Essentially attention for semseg model - channel-wise attention, spatial and mixed attention

- Paper arxiv.org/abs/1803.02579

- Implementation www.kaggle.com/c/tgs-salt-identification-challenge/discussion/66178

#deep_learning

snakers4 (Alexander), October 23, 06:28

DS/ML digest 27

NLP in the focus again!

spark-in.me/post/2018_ds_ml_digest_27

Also your humble servant learned how to do proper NMT =)

#digest

#deep_learning

#data_science

2018 DS/ML digest 27

2018 DS/ML digest 27 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), October 22, 05:43

Amazing articles about image hashing

Also a python library

- Library github.com/JohannesBuchner/imagehash

- Articles:

fullstackml.com/wavelet-image-hash-in-python-3504fdd282b5http://www.hackerfactor.com/blog/index.php?/archives/529-Kind-of-Like-That.html

www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html

www.hackerfactor.com/blog/index.php?/archives/529-Kind-of-Like-That.html

#data_science

#computer_vision

JohannesBuchner/imagehash

A Python Perceptual Image Hashing Module. Contribute to JohannesBuchner/imagehash development by creating an account on GitHub.


Text iterators in PyTorch

Looks like PyTorch has some handy data-processing / loading tools for text models - torchtext.readthedocs.io.

It is explained here - bastings.github.io/annotated_encoder_decoder/ - how to use them with pack_padded_sequence and pad_packed_sequence to boost PyTorch NLP models substantially.

#nlp

#deep_learning

snakers4 (Alexander), October 16, 03:47

Mixed precision distributed training ImageNet example in PyTorch

github.com/NVIDIA/apex/blob/master/examples/imagenet/main.py

#deep_learning

NVIDIA/apex

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch - NVIDIA/apex


snakers4 (Alexander), October 15, 17:11

Looks like mixed precision training ... is solved in PyTorch

Lol - and I could not find it

github.com/NVIDIA/apex/tree/master/apex/amp

#deep_learning

NVIDIA/apex

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch - NVIDIA/apex


snakers4 (Alexander), October 15, 09:33

DS/ML digest 26

More interesting NLP papers / material ...

spark-in.me/post/2018_ds_ml_digest_26

#digest

#deep_learning

#data_science

2018 DS/ML digest 26

2018 DS/ML digest 26 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), October 06, 13:04

PCIE risers that REALLY WORK for DL

Thermaltake TT Premium PCIE 3.0 extender.

All the others I tried were crap.

#deep_learning

snakers4 (Alexander), October 06, 07:24

Monkey patching a PyTorch model

Well, ideally you should not do this.

But sometimes you just need to quickly test something and amend your model on the fly.

This helps:

import torch

import functools

def rsetattr(obj, attr, val):

pre, _, post = attr.rpartition('.')

return setattr(rgetattr(obj, pre) if pre else obj, post, val)

def rgetattr(obj, attr, *args):

def _getattr(obj, attr):

return getattr(obj, attr, *args)

return functools.reduce(_getattr, [obj] + attr.split('.'))

for module in model.named_modules():

old_module_path = module[0]

old_module_object = module[1]

# replace an old object with the new one

# copy some settings and its state

if isinstance(old_module_object,torch.nn.SomeClass):

new_module = SomeOtherClass(old_module_object.some_settings,

old_module_object.some_other_settings)

new_module.load_state_dict(module_object.state_dict())

rsetattr(model,old_module_path,new_module)

The above code essentially does the same as:

model

.path.to.some.block = some_other_block

`

#python

#pytorch

#deep_learning

#oop

snakers4 (Alexander), October 02, 03:01

New release of keras

github.com/keras-team/keras/releases/tag/2.2.3

#deep_learning

keras-team/keras

Deep Learning for humans. Contribute to keras-team/keras development by creating an account on GitHub.


snakers4 (Alexander), September 28, 11:12

DS/ML digest 25

spark-in.me/post/2018_ds_ml_digest_25

#digest

#deep_learning

#data_science

2018 DS/ML digest 25

2018 DS/ML digest 25 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), September 20, 16:06

DS/ML digest 24

Key topics of this one:

- New method to calculate phrase/n-gram/sentence embeddings for rare and OOV words;

- So many releases from Google;

spark-in.me/post/2018_ds_ml_digest_24

If you like our digests, you can support the channel via:

- Sharing / reposting;

- Giving an article a decent comment / a thumbs-up;

- Buying me a coffee (links on the digest);

#digest

#deep_learning

#data_science

2018 DS/ML digest 24

2018 DS/ML digest 24 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), September 06, 06:18

SeNet

- arxiv.org/abs/1709.01507;

- A 2017 Imagenet winner;

- Mostly ResNet-152 inspired network;

- Transfers well (ResNet);

- Squeeze and Excitation (SE) block, that adaptively recalibratess channel-wise feature responses by explicitly modelling in- terdependencies between channels;

- Intuitively looks like - convolution meet the attention mechanism;

- SE block:

- pics.spark-in.me/upload/aa50a2559f56faf705ad6639ac973a38.jpg

- Reduction ratio r to be 16 in all experiments;

- Results:

- pics.spark-in.me/upload/db2c98330744a6fd4dab17259d5f9d14.jpg

#deep_learning

snakers4 (Alexander), September 06, 05:57

Chainer - a predecessor of PyTorch

Looks like

- PyTorch was based not only on Torch, but also its autograd was forked from Chainer;

- Chainer looks like PyTorch ... but not by Facebook, but by independent Japanese group;

- A quick glance through the docs confirms that PyTorch and Chainer APIs look 90% identical (both numpy inspired, but using different back-ends);

- Open Images 2nd place was taken by people using Chainer with 512 GPUs;

- I have yet to confirm myself that PyTorch can work with a cluster (but other people have done it) github.com/eladhoffer/convNet.pytorch;

www.reddit.com/r/MachineLearning/comments/7lb5n1/d_chainer_vs_pytorch/

docs.chainer.org/en/stable/comparison.html

#deep_learning

eladhoffer/convNet.pytorch

ConvNet training using pytorch. Contribute to eladhoffer/convNet.pytorch development by creating an account on GitHub.


Also - thanks for all DO referral link supporters - now finally hosting of my website is free (at least for next ~6 months)!

Also today I published a 200th post on spark-in.me. Ofc not all of these are proper long articles, but nevertheless it's cool.

snakers4 (Alexander), September 06, 05:48

DS/ML digest 23

The key topic of this one - is this is insanity

- vid2vid

- unsupervised NMT

spark-in.me/post/2018_ds_ml_digest_23

If you like our digests, you can support the channel via:

- Sharing / reposting;

- Giving an article a decent comment / a thumbs-up;

- Buying me a coffee (links on the digest);

Let's spread the right DS/ML ideas together.

#digest

#deep_learning

#data_science

2018 DS/ML digest 23

2018 DS/ML digest 23 Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), September 03, 06:27

Training a MNASNET from scratch ... and failing

As a small side hobby we tried training new Google's mobile network from scratch and failed:

- spark-in.me/post/mnasnet-fail-alas

- github.com/snakers4/mnasnet-pytorch

Maybe you know how to train it properly?

Also now you can upvote articles on spark in me! =)

#deep_learning

Training your own MNASNET

Training your own MNASNET Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), September 02, 06:22

A small hack to spare PyTorch memory when resuming training

When you resume from a checkpoint, consider adding this to save GPU memory:

del checkpoint

torch.cuda.empty_cache()

#deep_learning