Spark in me - Internet, data science, math, deep learning, philo

snakers4 @ telegram, 1317 members, 1587 posts since 2016

All this - lost like tears in rain.

Data science, deep learning, sometimes a bit of philosophy and math. No bs.

Our website
- spark-in.me
Our chat
- goo.gl/WRm93d
DS courses review
- goo.gl/5VGU5A
- goo.gl/YzVUKf

snakers4 (Alexander), July 21, 07:51

Yet another kaggle competition with high prizes and easy challenge

www.kaggle.com/c/tgs-salt-identification-challenge

#deep_learning

TGS Salt Identification Challenge

Segment salt deposits beneath the Earth's surface


snakers4 (Alexander), July 18, 05:39

Lazy failsafe in PyTorch Data Loader

Sometimes you train a model and testing all the combinations of augmentations / keys / params in your dataloader is too difficult. Or the dataset is too large, so it would take some time to check it properly.

In such cases I usually used some kind of failsafe try/catch.

But looks like even simpler approach works:

if img is None:

# do not return anything

pass

else:

return img

#deep_learning

#pytorch

snakers4 (Alexander), July 17, 08:51

Colab SeedBank

- TF is everywhere (naturally) - but at least they use keras

- On the other hand - all of the files are (at least now) downloadable via .ipynb or .py

- So - it may be a good place to look for boilerplate code

Also interesting facts, that are not mentioned openly

- Looks like they use Tesla K80s, which practically are 2.5-3x slower than 1080Ti

(medium.com/initialized-capital/benchmarking-tensorflow-performance-and-cost-across-different-gpu-options-69bd85fe5d58)

- Full screen notebook format is clearly inspired by Jupyter plugins

- Ofc there is a time limit for GPU scripts and GPU availability is not guaranteed (reported by people who used it)

- Personally - it looks a bit like slow instances from FloydHub - time limitations / slow GPU etc/etc

In a nutshell - perfect source of boilerplate code + playground for new people.

#deep_learning

Benchmarking Tensorflow Performance and Cost Across Different GPU Options

Machine learning practitioners— from students to professionals — understand the value of moving their work to GPUs . Without one, certain…


snakers4 (Alexander), July 17, 08:32

snakers4 (Alexander), July 16, 17:37

youtu.be/lCoR-4OlIZI

Style Transfer...For Smoke and Fluids! | Two Minute Papers #264
The paper "Example-based Turbulence Style Transfer" is available here: nishitalab.org/user/syuhei/TurbuStyleTrans/turbu_styletrans.html Pick up cool p...

snakers4 (Alexander), July 15, 08:54

Sometimes in supervised ML tasks leveraging the data sctructure in a self-supervised fashion really helps!

Playing with CrowdAI mapping competition

In my opinion it is a good test-ground for testing your ideas with SemSeg - as the dataset is really clean and balanced

spark-in.me/post/a-small-case-for-search-of-structure-within-your-data

#deep_learning

#data_science

#satellite_imaging

Playing with Crowd-AI mapping challenge - or how to improve your CNN performance with self-supervised techniques

In this article I tell about a couple of neat optimizations / tricks / useful ideas that can be applied to many SemSeg / ML tasks Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (spark_comment_bot), July 15, 06:16

Feeding images / tensors of different size using PyTorch dataloader classes

Struggled to do this properly on DS Bowl (I resorted to random crops there for training and 1-image sized batches for validation).

Suppose your dataset has some internal structure in it.

For example - you may have images of vastly different aspect ratios (3x1, 1x3 and 1x1) and you would like to squeeze every bit of performance from your pipeline.

Of course, you may pad your images / center-crop them / random crop them - but in this case you will lose some of the information.

I played with this on some tasks - sometimes force-resize works better than crops, but trying to apply your model convolutionally worked really good on SemSeg challenges.

So it may work very well on plain classification as well.

So, if you apply your model convolutionally, you will end up with differently-sized feature maps for each cluster of images.

Within the model, it can be fixed with:

(0) Adaptive avg pooling layers

(1) Some simple logic in .forward statement of the model

But anyway you end up with a small technical issue - PyTorch cannot concatenate tensors of different sizes using standard collation function.

Theoretically, there are several ways to fix this:

(0) Stupid solution - create N datasets, train on them sequentially.

In practice I tried that on DS Bowl - it worked poorly - the model overfitted to each cluster, and then performed poorly on next one;

(1) Crop / pad / resize images (suppose you deliberately want to avoid that);

(2) Insert some custom logic into PyTorch collattion function, i.e. resize there;

(3) Just sample images so that only images of one size end up within each batch;

(0) and (1) I would like to avoid intentionally.

(2) seems a bit stupid as well, because resizing should be done as a pre-processing step (collation function deals with normalized tensors, not images) and it is better not to mix purposes of your modules

Ofc, you can try to produce N tensors in (2) - i.e. tensor for each image size, but that would require additional loop downstream.

In the end, I decided that (3) is the best approach - because it can be easily transferred to other datasets / domains / tasks.

Long story short - here is my solution - I just extended their sampling function:

github.com/pytorch/pytorch/issues/1512#issuecomment-405015099

Maybe it is worth a PR on Github?

What do you think?

#deep_learning

#data_science

Like this post or have something to say => tell us more in the comments or donate!

[feature request] Support tensors of different sizes as batch elements in DataLoader #1512

Motivating example is returning bounding box annotation for images along with an image. An annotation list can contain variable number of boxes depending on an image, and padding them to a single length (and storing that length) may be n...


snakers4 (Alexander), July 14, 10:57

Once again stumbled upon this amazing PyTorch related post

For those learning PyTorch

discuss.pytorch.org/t/feedback-on-pytorch-for-kaggle-competitions/2252/11

#deep_learning

#pytorch

Feedback on PyTorch for Kaggle competitions

Hello team, Great work on PyTorch, keep the momentum. I wanted to try my hands on it with the launch of the new MultiLabeling Amazon forest satellite images on Kaggle. Note: new users can only post 2 links in a post so I can’t direct link everything I created the following code as an example this weekend to load and train a model on Kaggle data and wanted to give you my feedback on PyTorch. I hope it helps you. Loading data that is not from a regular dataset like MNIST or CIFAR is confusin...


snakers4 (Alexander), July 13, 17:05

Forwarded from Админим с Буквой:

Git commit messages

Как правильно комиттить в гит. Хорошая статья с хабра:

habr.com/post/416887/

#thirdparty #read #git

Как следует писать комментарии к коммитам

Предисловие от переводчика На протяжении многих лет разработки ПО, будучи участником многих команд, работая с разными хорошими и опытными людьми, я часто...


snakers4 (Alexander), July 13, 09:15

Tensorboard + PyTorch

6 months ago looked at this - and it was messy

now it looks really polished

github.com/lanpa/tensorboard-pytorch

#data_science

lanpa/tensorboard-pytorch

tensorboard-pytorch - tensorboard for pytorch (and chainer, mxnet, numpy, ...)


snakers4 (spark_comment_bot), July 13, 05:22

2018 DS/ML digest 17

Highlights of the week

(0) Troubling trends with ML scholars

approximatelycorrect.com/2018/07/10/troubling-trends-in-machine-learning-scholarship/

(1) NLP close to its ImageNet stage?

thegradient.pub/nlp-imagenet/

Papers / posts / articles

(0) Working with multi-modal data distill.pub/2018/feature-wise-transformations/

- concatenation-based conditioning

- conditional biasing or scaling ("residual" connections)

- sigmoidal gating

- all in all this approach seems like a mixture of attention / gating for multi-modal problems

(1) Glow, a reversible generative model which uses invertible 1x1 convolutions

blog.openai.com/glow/

(2) Facebooks moonshots - I kind of do not understand much here

- research.fb.com/facebook-research-at-icml-2018/

(3) RL concept flaws?

- thegradient.pub/why-rl-is-flawed/

(4) Intriguing failures of convolutions

eng.uber.com/coordconv/ - this is fucking amazing

(5) People are only STARTING to apply ML to reasoning

deepmind.com/blog/measuring-abstract-reasoning/

Yet another online book on Deep Learning

(1) Kind of standard livebook.manning.com/#!/book/grokking-deep-learning/chapter-1/v-10/1

Libraries / code

(0) Data version control continues to develop dvc.org/features

#deep_learning

#data_science

#digest

Like this post or have something to say => tell us more in the comments or donate!

Troubling Trends in Machine Learning Scholarship

By Zachary C. Lipton* & Jacob Steinhardt* *equal authorship Originally presented at ICML 2018: Machine


snakers4 (Alexander), July 12, 17:45

youtu.be/gnctSz2ofU4

DeepMind's AI Learns To See | Two Minute Papers #263
Pick up cool perks on our Patreon page: www.patreon.com/TwoMinutePapers Crypto and PayPal links are available below. Thank you very much for your gen...

snakers4 (Alexander), July 12, 05:11

Hadoop job in Moscow

No bullshit. The salary is net.

telegra.ph/Vakansiya-Junior--Middle--Senior-hadoop-developer-07-12

#jobs

Вакансия Junior / Middle / Senior hadoop developer

dilyara_tchk

Junior / Middle / Senior hadoop developer, 500-1500 р/час net (обсуждается по итогам собеседования).


На основе своего опыта могу сказать, что мы стараемся все делать гибко, быстро, с минимумом буллшита.

Ищем разработчика hadoop на участие в проекте в июне-декабре 2018 года. Возможно, и дальше. В команде имеется тимлид, админы.

В проекте имеется десятки источников больших данных, необходимо разрабатывать интеграции, производить расчет производных витрин.

Говоря простым языком, есть много данных из разных систем, их нужно положить к нам, потом посчитать на их основе что-то, снова посчитать (повторить необходимое количество раз), может быть, переложить по цепочке дальше. В отличие от стандартных ETL, фишка в экосистеме и в объеме данных.

В приоритете - работа full-time в офисе, но могут быть рассмотрены варианты удаленной работы.

Загрузка может варьироваться, 30-50 часов в неделю. Мы ожидаем, что это ваша основная работа. Совместительство с full-time работой не подходит.

Требования:

- понимание общей архитектуры hadoop

- git, хорошие практики разработки

- уверенная работа с Linux

- знание SQL (Hive)

- знание Spark (python)

Приветствуется:

- знакомство с HortonWorks Data Platform

- Java

Дополнительное пояснение: если вы джун, который хочет получить экспертизу в достаточно закрытом на текущий момент домене, в котором сейчас не хватает людей на рынке, это хороший шанс. От джуна требуется знание хотя бы python, умение в linux и git, инициативность и желание обучаться, в том числе самостоятельно.

Контакт: @dilyara_tchk, и нет, я не hr.


snakers4 (Alexander), July 11, 16:26

www.scholarcy.com/

Looks cool

But Mendeley so far looks better

Home

Scholarcy reads and summarises research papers, generates a background reading list, highlight important sentences, and finds open access versions of referenced papers.  


snakers4 (Alexander), July 11, 06:51

TF 1.9

github.com/tensorflow/tensorflow/releases/tag/v1.9.0

Funnily enough, they call Keras not "Keras with TF back-end", but "tf.keras"

xD

#deep_learning

tensorflow/tensorflow

An Open Source Machine Learning Framework for Everyone - tensorflow/tensorflow


snakers4 (Alexander), July 10, 04:35

Ofc such experiments are done on toy datasets - but it's nice to know

Forwarded from Just links:

Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks arxiv.org/abs/1806.10064

Forwarded from Hacker News:

NLP's ImageNet moment has arrived (Score: 100+ in 5 hours)

Link: readhacker.news/s/3MB3E

Comments: readhacker.news/c/3MB3E

NLP's ImageNet moment has arrived

The time is ripe for practical transfer learning to make inroads into NLP.


snakers4 (Alexander), July 09, 16:17

blog.bradfieldcs.com/an-introduction-to-hashing-in-the-era-of-machine-learning-6039394549b0

An Introduction to Hashing in the Era of Machine Learning

In December 2017, researchers at Google and MIT published a provocative research paper about their efforts into “learned index structures”…


snakers4 (Alexander), July 09, 09:04

2018 DS/ML digest 16

Papers / posts

(0) RL now solves Quake

venturebeat.com/2018/07/03/googles-deepmind-taught-ai-teamwork-by-playing-quake-iii-arena/

(1) A fast.ai post about AdamW

www.fast.ai/2018/07/02/adam-weight-decay/

-- Adam generally requires more regularization than SGD, so be sure to adjust your regularization hyper-parameters when switching from SGD to Adam

-- Amsgrad turns out to be very disappointing

-- Refresher article ruder.io/optimizing-gradient-descent/index.html#nadam

(2) How to tackle new classes in CV

petewarden.com/2018/07/06/what-image-classifiers-can-do-about-unknown-objects/

(3) A new word in GANs?

-- ajolicoeur.wordpress.com/RelativisticGAN/

-- arxiv.org/pdf/1807.00734.pdf

(4) Using deep learning representations for search

-- goo.gl/R1vhTh

-- library for fast search on python github.com/spotify/annoy

(5) One more paper on GAN convergence

avg.is.tuebingen.mpg.de/publications/meschedericml2018

(6) Switchable normalization - adds a bit to ResNet50 + pre-trained models

github.com/switchablenorms/Switchable-Normalization

Datasets

(0) Disney starts to release datasets

www.disneyanimation.com/technology/datasets

Market / interesting links

(0) A motion to open-source GitHub

github.com/dear-github/dear-github/issues/304

(1) Allegedly GTX 1180 start in sales appearing in Asia (?)

(2) Some controversy regarding Andrew Ng and self-driving cars goo.gl/WNW4E3

(3) National AI strategies overviewed - goo.gl/BXDCD7

-- Canada C$135m

-- China has the largest strategy

-- Notably - countries like Finland also have one

(4) Amazon allegedly sells face recognition to the USA goo.gl/eDzekn

#data_science

#deep_learning

Google’s DeepMind taught AI teamwork by playing Quake III Arena

Google’s DeepMind today shared the results of training multiple AI systems to play Capture the Flag on Quake III Arena, a multiplayer first-person shooter game. The AI played nearly 450,000 g…


snakers4 (Alexander), July 08, 17:47

Yet another proxy - shadowsocks

If someone needs another proxy guide, someone with an Arabic username shared some alternative advice for proxy configuration

- disq.us/p/1tsy4nk (wait a bit till link resolves)

#internet

#linux

Playing with a simple SOCKS5 proxy server on Digital Ocean and Ubuntu 16

This article tells you how to start your SOCKS5 proxy with zero to little experience Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), July 08, 06:16

Disclaimer - it does not support pivot tables or complicated group_by ...

snakers4 (Alexander), July 08, 06:06

A new multi-threaded addition to pandas stack?

Read about this some time ago (when this was just in development //snakers41.spark-in.me/1850) - found essentially 3 alternatives

- just being clever about optimizing your operations + using what is essentially a multi-threaded map/reduce in pandas //snakers41.spark-in.me/1981

- pandas on ray

- dask (overkill)

Links:

(0) rise.cs.berkeley.edu/blog/pandas-on-ray-early-lessons/

(1) www.reddit.com/comments/8wuz7e

(2) github.com/modin-project/modin

So...I ran a test in the notebook I had on hand. It works. More tests will be done in future.

pics.spark-in.me/upload/2c7a2f8c8ce1dd7a86a54ec3a3dcf965.png

#data_science

#pandas

Spark in me - Internet, data science, math, deep learning, philosophy

Pandas on Ray - RISE Lab https://rise.cs.berkeley.edu/blog/pandas-on-ray/


snakers4 (Alexander), July 08, 04:52

youtu.be/FwFduRA_L6Q

Convolutional Network Demo from 1993
This is a demo of "LeNet 1", the first convolutional network that could recognize handwritten digits with good speed and accuracy. It was developed between 1...

snakers4 (spark_comment_bot), July 07, 12:29

Playing with VAEs and their practical use

So, I played a bit with Variational Auto Encoders (VAE) and wrote a small blog post on this topic

spark-in.me/post/playing-with-vae-umap-pca

Please like, share and repost!

#deep_learning

#data_science

Like this post or have something to say => tell us more in the comments or donate!

Playing with Variational Auto Encoders - PCA vs. UMAP vs. VAE on FMNIST / MNIST

In this article I thoroughly compare the performance of VAE / PCA / UMAP embeddings on a simplistic domain - UMAP Статьи автора - http://spark-in.me/author/snakers41 Блог - http://spark-in.me


snakers4 (Alexander), July 07, 08:27

youtu.be/qS4H6PEcCCA

Epicycles, complex Fourier series and Homer Simpson's orbit
Today’s video was motivated by an amazing animation of a picture of Homer Simpson being drawn using epicycles. This video is about making sense of the mathem...

snakers4 (Alexander), July 06, 15:33

Forwarded from Админим с Буквой:

Bash shortcuts

Написал микро лабораторную работу для обучения хоткеям в bash.

medium.com/@bykvaadm/bash-shortcuts-d6f275a6ce9d

#bash_tips_and_tricks #junior

bash shortcuts

Небольшая лабораторка по изучению основных хоткеев в bash. Подготовьте себе вот такую строку:


snakers4 (Alexander), July 05, 05:54

XGB - now on GPU properly?

twitter.com/i/web/status/1014192185510629378

Joshua Patterson

#XGBoost is faster than ever, with better scaling, on #GPU thanks to the hard work of @nvidia & @h2oai! Check out the latest paper https://t.co/P2m31idljB, and more is coming very soon! #lightgbm #catboost #GBDT


snakers4 (Alexander), July 04, 16:56

Forwarded from Just links:

twitter.com/Foone/status/1014267515696922624

foone

You want to know something about how bullshit insane our brains are? OK, so there's a physical problem with our eyes: We move them in short fast bursts called "saccades", right? very quick, synchronized movements. The only problem is: they go all blurry and useless during this


snakers4 (Alexander), July 04, 14:16

www.youtube.com/watch?utm_campaign=Revue+newsletter&utm_medium=Newsletter&utm_source=NLP+News&v=3o4VzEyJ0WA

Machine Learning Research & Interpreting Neural Networks
Machine learning and neural networks change how computers and humans interact, but they can be complicated to understand. In this episode of Coffee with a Go...

snakers4 (Alexander), July 04, 08:11

Forwarded from Savva Kolbachev:

nlp.town/blog/sentence-similarity/

Comparing Sentence Similarity Methods

Natural Language Processing Consultancy and Development.


snakers4 (Alexander), July 04, 07:57

2018 DS/ML digest 15

What I filtered through this time

Market / news

(0) Letters by big company employees against using ML for weapons

- Microsoft

- Amazon

(1) Facebook open sources Dense Pose (eseentially this is Mask-RCNN)

- research.fb.com/facebook-open-sources-densepose/

Papers / posts / NLP

(0) One more blog post about text / sentence embeddings goo.gl/Zm8C2c

- key idea different weighting

(1) One more sentence embedding calculation method

- openreview.net/pdf?id=SyK00v5xx ?

(2) Posts explaing NLP embeddings

- www.offconvex.org/2015/12/12/word-embeddings-1/ - some basics - SVD / Word2Vec / GloVe

-- SVD improves embedding quality (as compared to ohe)?

-- use log-weighting, use TF-IDF weighting (the above weighting)

- www.offconvex.org/2016/02/14/word-embeddings-2/ - word embedding properties

-- dimensions vs. embedding quality www.cs.princeton.edu/~arora/pubs/LSAgraph.jpg

(3) Spacy + Cython = 100x speed boost - goo.gl/9TwVqu - good to know about this as a last resort

- described use-case

you are pre-processing a large training set for a DeepLearning framework like pyTorch/TensorFlow

or you have a heavy processing logic in your DeepLearning batch loader that slows down your training

(4) Once again stumbled upon this - blog.openai.com/language-unsupervised/

(5) Papers

- Simple NLP embedding baseline goo.gl/nGujzS

- NLP decathlon for question answering goo.gl/6HHi7q

- Debiasing embeddings arxiv.org/abs/1806.06301

- Once again transfer learning in NLP by open-AI - goo.gl/82VR4U

#deep_learning

#digest

#data_science

Download full.pdf 0.04 MB