Hidden pearls for high-performance-persistence in Java

Presented by Sven Ruppert - DevSecOps from JFrog.

Presentation was pre-recorder, but the questions were live.

Why and what: Inside of JVM, for different areas - from embedded, to terabyte-big document storage systems, but scaling and tuning ways of storing data.

You can’t get rid of the persistence layer, but there are three solutions as the beginning of making optimization.

JOOQ: a tool for mapping SQL to Java DSL - SQL style, generated from the DB schema. Allows doing create.select(BOOK.title).from(BOOK).where(...).orderBy(...). Lifecycle: generate Java classes based on SQL schema, can work with it from there.

Image from Sven Rupperts’ presentation

Speedment: streams over JDBC, generated from the DB schema, generating Java classes with streaming API.

(Another tool: Querry DSL)

JPA streamer built on top of hibernate, same stream API.

Chronicle Bytes: fast solution for writing data down. Provides two classes: Bytes - a new way for creating bytes buffers. MappedBytes - mapping buffer to file. Inspired by C++, easy way to define some binary data files, loose description of classes/fields, etc, have to keep track on your own.

Chronicle Maps: fast JVM key-value store, have to think about serialization for your own. ChronicleMap<LongValue, CharSequence>.

XODUS: transactional schema-less embedded database used by JetBrains YouTrack and JetBrains Hub. (Transactional, ACID, highly concurrent, embedded, open-source DB). Snapshot isolation only - all changes are written sequentially to a log, log is immutable => persistent functional data structure => Garbage Collector is needed.

Environment -> create instance (directory) -> inside env. create and run transactions.

Key-value layer, entity store layer, virtual file system layer.

MapDB combines embedded database engine and Java collections. Can be used as a drop-in replacement for Maps, Lists, Queues, off-heap collections not affected by GC, multilevel cache with expiration and disk overflow, RDBMs replacement with transactions, MVCC, incremental backup, local data processing, and filtering, etc.

DB db = DBMaker.fileDB("/some/file").encryptionEnable("password").make();, ConcurrentNavigatableMap<Integer, String> map = db.treeMap(...).

You can create an in-memory map, a on-disk map and combine them (for example, for overflowing). Possible to build cascaded data-structures with different attributes.

Microstream - opensource tool, with custom serialization for objects, not the standart Java one. Good for working with graphs.

Start with EmbeddedstoreManager, define root, store root, call shutdown(). Store model as it is, store collections, can load partial graphs, even cycles are not a problem, no special inheritance.

Advertizes MicroStream hackaton - show exciting use-cases and how MicroStream integrates with any Framework or Library.

Mentioned backup/restore functionality in all those projects, advertized own blog. Mentioned security, dealing with vulnerabilities, shift left testing. Mentions security payload generator injection.

Workshop: Stop writing tests, write specs instead!

Presented by Alexey Nesterov, experts: Anatoliy Korovin, Timur Nasredinov

Speaker had a lot of experience with TDD development process. “Why write testing code for something, that isn’t created yet?” - you don’t. You write specifications instead, you ask your code “What if we do this? What if we do what?”

On benefits of unit tests and BDD:

“Refactoring without tests is just moving shit around”

If you write tests after writing code - you can’t be sure if they can really break

TDD approach helps with task decomposition

Speaker’s view on TDD: “Test Driven Development Design”

Red-Green-Refactor. Don’t overengineer! Don’t forget to refactor - that’s important! The easier it is to start running tests, the more pleasurable development will be.

P(tests) = 1 / manual-steps

(The more manual steps you need to take to run tests - the fewer chances they will be run!)

<Demo - test with lots of actions, the speaker asked what’s wrong with it; experts: too many actions?; speaker: unclear what is being tested, may fail in a lot of different ways, unclear behavior of external api>

What is unit [in Spring app] test? Component? Component + Controller? Several communicating components? Typically it should be a logically isolated block.

Acceptance/E2E: from users’ perspective, no internal system knowledge, typically - happy paths, slow and unstable. <- BDD lives here.

Forget about coverage - it’s useless. If all tests being green is enough for you to go to production - assume coverage to be 100%.

Workshop starts: test example - pet store, need to add functionality to display some animals on a front page.

Presenter starts with setting up sample db with just id and name for animal table, writes test “Send request, it returns OK”. This forces us to create a controller, add some sort of get mapping.

Different “schools” of thought - minimum mocks, real (staging) DBs, or mock everything?

Proceed by adding first spring component with no code, mocking it, writing GET test;

@SpringBootTest - test full app; @DataJdbcTest (or other slices) - test only part of service layers.

Do not mock Spring framework components, like Repository or Spring Security - this can lead to green tests and failing environment.

Run E2E tests in as realistic environment as possible!

Novel but practical uses of Java

Presented by Peter Lawrey from OpenHFT.

Extending Stack Trace to track where object was created, and where the object was closed. Report where threads are still running. Was tested in Java 14

(Use resources for long period of time, can’t rely on try-with-resources, have to find own solution. Have background releaser thread - use this for testing. )

DynamicEnum - want to work by name, but have possibility to add new values as well.

Fixed precision rounding: round2()

MachineUniqueMicroTimeProvider;

StringInterner - a collection without explicit thread safety, that still behaves safely.

Custom IllegalStateException - By adding it into a classpath before system libraries - the compiler now thinks they are checked exceptions. Could be substituted with code analyser.

Lambda friendly ClassValue - associate a piece of data with a class => that data is retired when class is unloaded.

Treating checked exceptions as unchecked:

public static <T extends Throwable> RuntimeException retrow(Throwable throwable) throws T { throw (T) throwable;} (Jvm.rethrow());

public static <T extends Throwable> RuntimeException rethrow(Throwable t) throws T {
    throw (T) t;
  }

  @FunctionalInterface
  public interface ThrowingConsumer<I, T extends Throwable> {
    void accept(I in) throws T;
  }

  static <I, T extends Throwable> Consumer<I> asConsumer(ThrowingConsumer<I, T> function) {
    return in -> {
      try {
        function.accept(in);
      } catch (Throwable t) {
        rethrow(t);
      }
    };
  }

  public void a() {
    asConsumer(v -> scoringOverduesComponent.loadAndSaveOverdues(userProfile, userProfile.getLoginCountry(), subSurveyNbr, LicenseType.BANK))
      .accept(null);
  }

Safepoints Java?

Spring Patterns for adults

Energetically presented by Evgeny Borisov, experts: Andrei Kogun, Kirill Tolkachev.

Practical presentation, not some esoteric stuff. No BeanPostProcessros!

Singletons - great as conception, terrible as a pattern. Latest Java-supported solution - write Singleton as enum, can also in static block.

Singleton is an anti-pattern because it forces tight coupling, encourages writing ugly code, can’t be tested without PowerMock, defies Single Responsibility principle (it manages itself and it’s its’ own container).

Declaring singleton: use Spring.

@ComponentScan => starts scouting defined @Components (including @Service/@Repository/@Controller/@RestController. @Configuration - is a component too!) This works because most Spring mechanisms don’t just search for a specific annotation, but rather analyze each found annotation - what’s a superclass for it, if it’s something that Spring supports?

@Bean - ?

Lazy singleton: @Lazy. @Lazy @Component, @Lazy @Service…

<Question from expert about @Repository>. Presenter clarifies that you can make your own service, inject it with EntityManager or JdbcConnection, do some work with it, but adding @Repository wraps it up in a layer, that throws standard Spring exceptions that you’ll know how to handle. (Or to test only DB layer)

: Autowiring @Lazy component forces Spring to create that component from the very beginning - Spring wouldn’t know otherwise when you will use it. To solve this issue use @Autowired @Lazy - will inject some proxy that will create component and delegate call only when needed.

Lazy injection - since Spring 4.3.

Tests

Building full context, or only parts of context? Testing parts of the system => (Something breaks, I don’t know why) => Using main conf => All beans are now created, not only related to test scenario.

Solution: @ComponentScan(lazyInit = true) - will create full context, but lazily. In application.properties => spring.main.lazy-initialization=true.

“Constructor injection encourages developers to minimal classes?” Uh-huh, sure, @RequiredArgsConstructor laughs in your face. Secondly - Idea supports field injection better.

@Primary usages - don’t use it for defining mock beans. Can be used for qualifying what you inject, e.g. @Bean @Primary public RestTemplate restTemplate() -> @Autowired RestTemplate restTemplate (when you have several RestTemplate’s).

Image from Evgeny Borisovs’ presentation

Implementing Chain of responsibility: Spring knows how to autowire lists of something @Autowired List<Handler> handlers -> Handler1.java, Handler2.java, ....

Injecting custom lists (only this and that, or ensure order)? Use BeanPostProcessor (whoops!), to make a list out of ApplicationContext

Introspector.decapitalize() - used for making bean names.

Learn more about spring.factories.

Reusing old/legacy “beans” in a new Spring Boot application

(using Reflections library that extends java.reflection)

class LegacyBeanDefinitionRegistrar implements ImportBeanDefinitionRegistrar 

registerBeanDefinition() {
      Reflections scanner = new Reflections("com.legacy.package");
      Set<Class<?>> classes = scanner.getTypesAnnotatedWith(LegacySingleton.class);
      for(Class<?> aClass : classes) {
        GenericBeanDefinition bd = new G...();
        bd.setBeanClass(aClass);
        beanDefinition.setScope(beanDefinition.SCOPE_SINGLETON);

        registry.registerBeanDefinition(Introspector.decapitalize(aClass.getSimpleName()), bd);
      }
}

To add some other qualifier: bd.addQualifier(new AutowireCandidateQualifier(Legacy.class));

Start using: @Configuration @Import(LegacyBeanDefinitionRegistrar.class) in Configuration class.

More patterns:

Strategy & Command

Never use switch! “What’s 235 switch cases, when you already had 234 cases?” -> “Instead of digging shitty code, write as if you are already a senior dev!”

Option 1: Rename bean ids (bad example)

  @Autowired private Map<String, Something> map; // String = bean id

Option 2: add some key information to interface

interface SomethingInterface { String myType(); void doWork(); }


public class Controller {
  private Map<String, Something> map;

  public Controller()(List<SomethingInterface> somethings) {
    map = somethings.stream.collect(toMap(SomethingInterface::myType), Function.identity());
  }

  ...
}

Option 3 define Map<…> as bean, group it as in Option 2.

Registering beans on the fly:

1. Extend ClassLoader, load class from byte[]
2. call BeanDefinitionRegistry, create GenericBeanDefinition, register bean, call it from context.

Since Spring 4.3: @Autowired default void regMe(Controller c) {c.register(this.getType, this); } - Spring does not care if it’s a “get/set” or something another method - upon registering this bean (all superclasses of the interface), will call this method as if it was setter, and will pass Controller as parameter.

Don’t use AspectJ for business logic - you will be hated by everyone! You can still use AspectJ on a framework level, for example - to send warning notifications, every time ClassCastException happens.

Static approach:

@Aspect public class ExceptionHandlerAspect {
  @AfterThrowing(pointcut = "execution(*..*.*(..))", throwing = "ex")
  public void handleException(ClassCastException ex) {
    System.out.println("Something broke again!");
  }
}

Dynamic approach:

public class ExceptionHandlerAspect implements MethodInterceptor {
  public Object invoke(MethodInvocation methodInvocation) throws Throwable {
    try {
      return methodInvocation.proceed()
    } catch (ClassCastException ex) {
      System.out.println("Something broke again!");
    }
  }
}

public class CustomPointcut extends DynamicMethodMatcherPointcut {
  private String packagesToHandle = "...";

  public boolean matches(...) {
    return true;
  }

  public ClassFilter getClassFilter() {
    return .getPackage().getName().contains(packagesToHandle);
  }
}

... 
@Bean public DefaultPointcutAdvisor dfa() { return new DefaultPointcutAdvisor(customPointcut(), exceptionHandlerAspect()); }
...

Morale. Patterns which are simple easy with Spring:

Singleton (Lazy/Eager)

Chain of responsibility

Strategy

Command

Registry

Observer

Proxy

Spring techniques and mechanisms:

Lazy scanning

Primary / Qualifier / Bean name

Custom injection - BeanPostProcessor

Declaring beans from other framework - ImportBeanDefinitionREgistrar

Bean Registration after context was refreshed

Dynamic pointcuts

AOP from controllers - ConrollerAdvice

Think about it:

Extending Spring possibilities with custom annotations, conventions

Correct use of AOP

Default methods + Spring annotations = new options for interfaces

“My starter is always with me!”

Will robots replace programmers?

Presented by Tagir Valeev

<I skipped the first part of the presentation, can judge from Twitter comments that I missed a lot of fun>

16 Sep 2019 - first fully automated PR, created by robots, checked and approved by robots, and merged by robots (bump dependency version).

Diffblue - AI for auto-generation of unit tests.

GPT-3 - could potentially translate text specifications to programming code.

Tagir is still rather optimistic and sees AI (and IDE tools) development as a future for pair programming - some parts can be done by a robot, some parts - by a human, work together to get results faster.